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Progress  Made  Toward  Increased  Stability  under  USAID’s 
Afghanistan  Stabilization  Initiative-East  Program  but  Transition 
to  Long  Term  Development  Efforts  Not  Yet  Achieved 

What  SIGAR  Reviewed 

In  June  2009,  the  U.S.  Agency  for  International  Development’s  (USAID)  Office  of  Transition  Initiatives  (OTI)  awarded  a  3  year, 

$151  million  task  order  to  Development  Alternatives,  Inc.,  (DAI)  to  implement  the  Afghanistan  Stabilization  Initiative-East  (ASI- 
East)  program  in  Afghanistan.  ASI-East  focuses  on  awarding  grant  activities  to  local  implementers  in  10  districts  distributed  across 
four  provinces  in  eastern  Afghanistan.  A  key  objective  of  ASI-East  is  to  support  the  “hold”  phase  of  the  U.S.  Counterinsurgency 
(COIN)  strategy,  which  consists  of  “clear,  hold,  and  build”  phases  designed  to  permanently  stabilize  targeted  districts  and 
communities.  After  task  order  award,  OTI  asked  DAI  to  develop  a  more  strategic  approach  to  stabilization  programming  designed  to 
target  specific  sources  of  instability  at  the  district  level.  DAI  began  to  implement  this  new  approach,  the  District  Stability  Framework 
(DSF),  in  February  2010.  In  February  2012,  OTI  awarded  a  new  3-year,  $161.5  million  follow-on  task  order  to  Creative  Associates 
International,  Inc.,  to  continue  ASI  programming  efforts  in  eastern  and  southern  Afghanistan  under  a  single  implementing  partner. 

This  report  assesses  to  what  extent  (1)  expended  ASI-East  costs  were  allowable,  allocable,  and  reasonable;  (2)  performance  oversight, 
monitoring,  and  evaluation  systems  have  been  implemented;  and  (3)  progress  has  been  made  toward  transitioning  districts  to  the 
“build”  phase  of  the  COIN  strategy.  To  accomplish  these  objectives,  we  interviewed  Afghanistan-based  staff  from  USAID  OTI, 
USAID’s  Stabilization  Unit,  DAI,  and  third-party  monitoring  staff  hired  through  DAI  to  provide  independent  monitoring  and 
evaluation  of  the  program.  We  also  interviewed  USAID  OTI  staff  based  in  Washington,  D.C.  We  reviewed  the  ASI-East  task  order, 
related  procurement  documents,  and  a  sample  of  20  grants.  We  reviewed  a  range  of  performance  management  documents,  data,  and 
systems.  We  reviewed,  analyzed,  and  tested  select  DAI  billing  and  timekeeping  policies  and  procedures  and  incurred  costs  to 
determine  if  program  costs  were  allowable,  allocable,  and  reasonable.  We  conducted  work  in  Kabul,  Afghanistan,  and  Washington, 
D.C.,  from  May  201 1  to  June  2012  in  accordance  with  generally  accepted  government  auditing  standards. 


What  SIGAR  Found 

SIGAR  found  that  although  incurred  costs  under  the  ASI-East  task  order  were  generally  allowable,  allocable,  and  reasonable,  certain 
cost-related  issues  need  to  be  addressed.  These  issues  included  program  spending  that  was  marked  by  high  operating  costs,  more  than 
$590,000  in  questionable  program  costs,  and  a  limited  number  of  timekeeping  and  billing  deficiencies,  which  increased  the  risk  of 
inappropriate  charges. 

DAI  implemented  a  range  of  program  oversight,  monitoring,  and  evaluation  systems;  however,  final  program  results  remain  to  be 
determined,  and  certain  administrative  issues  relating  to  program  oversight  need  to  be  addressed  in  the  follow-on  task  order  award. 
USAID’s  efforts  to  evaluate  ASI-East’ s  final  program  results  will  be  critical  to  assisting  with  short  and  long-term  decisions  regarding 
whether  and  how  extensively  the  DSF  methodology  should  be  implemented  in  stabilization  efforts,  such  as  USAID’s  Stabilization  in 
Key  Areas  program.  Program  monitoring  and  evaluation  centered  on  the  three-tier  system  called  for  by  the  DSF  methodology. 
Preliminary  results  indicate  that  ASI-East  activities  have  been  successfully  implemented  and  that  the  impact  of  this  program  on 
stability  at  the  district  level  has  been  positive,  but  overall  stability  remains  poor  across  ASI-East’ s  10  programming  districts  based  on 
seven  leading  indicators  of  stability  developed  by  DAI  and  its  monitoring  and  evaluation  subcontractor. 

Despite  nearly  3  years  of  program  efforts,  none  of  ASI-East’ s  target  districts  have  transitioned  to  the  “build”  phase  of  the  COIN 
strategy,  which  is  designed  to  solidify  the  gains  during  the  “hold”  phase  of  COIN  operations.  OTI  has  only  recently  drafted  district- 
level  disengagement  criteria.  An  exit  strategy  for  OTI  programming  in  Afghanistan  remains  to  be  developed  under  the  follow-on  task 
order  for  ASI.  These  efforts  will  need  to  be  integrated  with  planned  improvements  and  evaluations  of  the  DSF  methodology. 

What  SIGAR  Recommends 

SIGAR  recommends  that  USAID  OTI  (1)  review  the  balance  between  project  and  operating  costs  resulting  from  the  application  of  the 
DSF  methodology  to  ASI-East  programming  decisions,  (2)  address  identified  timekeeping  and  billing  deficiencies,  (3)  review  the 
questioned  costs  identified  by  SIGAR,  (4)  include  outcome  indicators  under  the  DSF,  (5)  produce  a  lessons  learned  summary  of  ASI- 
East’s  implementation  of  the  DSF,  and  (6)  develop  approved  district-level  disengagement  criteria  and  a  related  exit  strategy  for  OTI 
programming  in  Afghanistan.  When  preparing  the  final  report,  SIGAR  considered  comments  from  USAID.  USAID  generally 
concurred  with  the  recommendations  and  noted  the  actions  it  has  taken  or  will  take  to  address  SIGAR’ s  recommendations.  Based  on 
USAID’s  comments,  SIGAR  revised  recommendations  four  and  five  to  also  address  the  Stabilization  Unit  Chief.  Through  its 
Measuring  Impact  of  Stabilization  Initiatives  program,  USAID’s  Stabilization  Unit  in  Kabul  is  responsible  for  monitoring  and 
evaluating  USAID”s  stabilization  efforts,  including  USAID’s  implementation  of  the  DSF. 


For  more  information  contact:  SIGAR  Public  Affairs  at  (703)  545-5974  or  sigar.pentagon.ccr.mbx.public-affairs@mail.mil 


SIGAR 


Special  Inspector  General  for  Afghanistan  Reconstruction 
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This  report  discusses  the  results  of  the  Office  of  the  Special  Inspector  General  for  Afghanistan 
Reconstruction’s  (SIGAR)  audit  of  the  U.S.  Agency  for  International  Development’s  (USAID)  task  order 
with  Development  Alternatives,  Inc.,  to  implement  the  Afghanistan  Stabilization  Initiative-East  program. 
This  report  includes  six  recommendations  to  USAID  that  address  a  range  of  issues  including  questioned 
costs,  programming  and  evaluation  methodology,  and  transition  planning. 

When  preparing  the  final  report,  we  considered  comments  from  USAID’s  Office  of  Transition  Initiatives. 
These  comments  are  reproduced  in  appendix  III.  SIGAR  conducted  this  performance  audit  under  the 
authority  of  Public  Law  No.  1 10-181,  as  amended,  the  Inspector  General  Act  of  1978,  and  the  Inspector 
General  Act  of  2008. 


Steven  J  Trent 

Acting  Special  Inspector  General 
for  Afghanistan  Reconstruction 
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Progress  Made  Toward  Increased  Stability  under  USAID’s  Afghanistan 
Stabilization  Initiative-East  Program  but  Transition  to  Long  Term 
Development  Efforts  Not  Yet  Achieved 


A  key  objective  of  the  Afghanistan  Stabilization  Initiative-East  (ASI-East)  is  to  support  the  coalition 
Counterinsurgency  (COIN)  strategy,  which  includes  “clear,  hold,  and  build”  phases  designed  to 
permanently  stabilize  target  districts  and  communities  in  areas  previously  controlled  by  insurgents.1  The 
ASI-East  program  focuses  on  implementing  small-scale  community  development  and  media  grants  in  ten 
districts  distributed  across  four  provinces  in  eastern  Afghanistan  during  the  “hold”  phase  of  the  strategy, 
which  occurs  before  long-term  development  efforts  are  initiated  in  the  “build”  phase.2  ASI-East  projects 
are  intended  to  build  allegiance  and  confidence  between  local  communities  and  the  Government  of  the 
Islamic  Republic  of  Afghanistan  (GIRoA)  in  target  regions  and  districts.  The  intended  outcome  is  to 
achieve  immediate  employment  generation,  improve  the  community’s  infrastructure,  and  increase  access 
to  public  services  with  the  overall  goal  of  improving  community  and  government  capacity  and  public 
support  for  GIRoA. 

USAID’s  Bureau  for  Democracy,  Conflict,  and  Humanitarian  Assistance/Office  of  Transition  Initiatives 
(DCHA/OTI)  implements  the  ASI  program  in  Afghanistan.  OTI  works  on  the  ground  in  countries  to 
provide  fast,  flexible,  short-term  assistance  targeted  at  key  political  transition  and  stabilization  needs. 

OTI  works  closely  with  USAID  regional  bureaus  and  missions  and  with  other  U.S.  counterparts  to 
identify  programs  that  complement  other  assistance  efforts  and  lay  a  foundation  for  longer-term 
development.  In  June  2009,  OTI  awarded  a  task  order  to  Development  Alternatives,  Inc.  (DAI)  under 
USAID’s  global  Support  Which  Implements  Fast  Transition  III  (SWIFT  III)  indefinite  quantity  contract 
(IQC).  The  ASI-East  program  is  a  3 -year  task  order  with  a  program  ceiling  of  $151  million.  After  task 
order  award,  OTI  asked  DAI  to  develop  a  more  strategic  approach  to  stabilization  programming  designed 
to  target  specific  sources  of  instability  at  the  district  level.  DAI  began  to  implement  this  new  approach, 
the  District  Stability  Framework  (DSF),  in  February  2010.  The  DSF  approach  represents  a  shared 
terminology  and  a  unified  assessment  process  for  identifying  and  resolving  sources  of  instability  in  target 
areas. 

This  report  assesses  to  what  extent  (1)  expended  ASI-East  costs  were  allowable,  allocable,  and 
reasonable;  (2)  performance  oversight,  monitoring,  and  evaluation  systems  have  been  implemented;  and 
(3)  progress  has  been  made  toward  transitioning  districts  to  the  “build”  phase  of  the  COIN  strategy. 

To  accomplish  these  objectives,  we  interviewed  staff  in  Afghanistan  from  USAID  OTI,  USAID’s 
Stabilization  Unit,  DAI,  and  third-party  contractor  staff  hired  by  DAI  to  provide  independent  monitoring 
and  evaluation  of  the  program.  We  also  interviewed  USAID  OTI  staff  in  Washington,  D.C.  including  the 


!The  COIN  strategy  includes  working  closely  with  Afghan  counterparts  to  rid  target  areas  of  insurgents  (“clear” 
phase),  provide  humanitarian  relief  and  initial  government  services  after  it  is  cleared  (“hold”  phase),  and  applying 
coordinated  civil-military  development  efforts  to  reinforce  security  gains  (  “build”  phase).  The  ASI-E  program  aims 
to  support  the  transition  from  “hold”  to  “build”  by  working  with  government  officials  and  through  traditional 
community  leadership  structures  to  identify  sources  of  instability  and  implement  projects  that  reinforce  security 
gains  made  by  coalition  forces.  These  projects  lay  the  groundwork  for  long-term  USAID  development  projects. 

2  Throughout  this  report,  we  refer  to  1 0  districts  when  discussing  ASI-East  program  activity.  This  number  of 
districts  was  accurate  as  of  October  2011  and  these  districts  do  represent  the  core  of  ASI-East  programming  efforts. 
Through  an  ongoing  assessment  process,  however,  OTI  made  the  decision  to  exit  and  enter  districts  at  multiple 
points  in  the  program.  As  of  May  2012,  the  ASI-East  program  had  active  projects  in  15  districts  in  Regional 
Command-East  and  3  districts  in  Regional  Command- South. 
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cognizant  contracting  officer  and  contracting  officer’s  technical  representative  (COTR)  for  the  ASI-East 
task  order.  We  reviewed  the  ASI-East  task  order  and  related  procurement  documents  including  the 
SWIFT  III  IQC.  We  reviewed  a  range  of  performance  management  documents,  data,  and  systems 
including  DATs  annual  and  quarterly  performance  reports,  activity  tracking  systems,  and  a  three-tier 
monitoring  and  evaluation  system  and  related  reports  and  studies.  We  reviewed,  analyzed,  and  performed 
tests  of  select  DAI  timekeeping  and  billing  policies  and  procedures  and  incurred  costs  to  determine  if 
program  costs  were  allowable,  allocable,  and  reasonable  per  Federal  Acquisition  Regulation  (FAR) 
principles.3  We  reviewed  DATs  grant  making  and  oversight  procedures  for  a  sample  of  20  completed 
and  closed  grants  awarded  under  the  task  order.  We  also  attended  the  3-day  DSF  training  course 
delivered  by  the  COIN  academy  with  support  from  trainers  provided  by  OTI  to  gain  a  deeper 
understanding  of  the  DSF  as  a  new  stabilization  programming  tool.  We  conducted  our  work  in  Kabul, 
Afghanistan,  and  Washington,  D.C.,  from  May  201 1  to  June  2012  in  accordance  with  generally  accepted 
government  auditing  standards. 


BACKGROUND 

The  ASI-East  task  order  required  DAI  to  meet  a  phased  program  schedule  covering  pre-deployment/start 
up,  full  implementation,  and  project  close  out.  DAI  project  close  out  efforts  have  not  yet  begun  because 
OTI  recently  extended  ASI-East’s  performance  period  by  3  months  to  September  2012.  This  extension 
was  designed  to  allow  a  more  orderly  transition  to  a  follow-on  task  order  recently  awarded  by  OTI.  The 
intent  of  the  follow-on  task  order  is  to  continue  the  ASI-East  and  parallel  programming  efforts  in  ASI- 
South  under  a  single  task  order  implemented  by  one  implementing  partner.  The  new  task  order  was 
awarded  to  Creative  Associates  International,  Inc.,  in  February  2012,  with  a  ceiling  price  of 
$161.5  million.  The  new  task  order  has  an  effective  start  date  of  March  2012  and  an  estimated 
completion  date  of  February  2015. 

With  regard  to  performance  expectations,  the  ASI-East  task  order  has  two  objectives:  (1)  create 
conditions  that  build  confidence  between  communities  and  GIRoA  through  improvement  of  the  economic 
and  social  environment  in  the  region,  and  (2)  increase  public  access  and  awareness  to  information  about 
GIRoA’s  social,  economic,  and  political  activities  and  policies  in  Afghanistan.  By  design,  the  task  order 
requires  DAI  to  be  responsive  to  changes  in  program  strategy  by  expanding,  decreasing,  or  changing  its 
program  activities  with  as  little  as  a  few  days  notice.  Consequently,  these  two  objectives  did  not  have 
associated  performance  goals  and  targets  because  the  task  order  was  viewed  by  OTI  as  a  tactical 
programming  tool  and  program  direction  might  need  to  change  on  very  short  notice. 

At  the  time  of  award,  DAI  used  the  military  Tactical  Conflict  Assessment  and  Planning  Framework  tool 
to  identify  and  select  stabilization  projects  for  the  ASI-East  program.  The  framework  sought  the 
perceptions  of  local  nationals  through  the  use  of  a  survey  to  determine  specific  sources  of  instability. 
Under  this  approach,  DAI  awarded  49  initial  grants  under  the  ASI-East  program  totaling  $1.3  million.  In 
December  2010,  through  the  ASI-East  task  order,  OTI  implemented  DSF  to  identify  specific  sources  of 
instability  and  program  resources  accordingly.  Through  October  2011,  DAI  awarded  an  additional  412 
grants  using  the  DSF  methodology  at  a  total  additional  cost  of  about  $21  million. 


2 

Costs  are  allowable  if  they  are  determined  to  be  allocable  and  reasonable,  among  other  factors.  A  cost  is  allocable 
to  a  Federal  award  if  it  is  treated  consistently  with  other  costs  incurred  for  the  same  purpose  in  like  circumstances, 
amongst  other  factors.  A  cost  is  considered  reasonable  if  in  its  nature  or  amount  it  does  not  exceed  that  which  would 
be  incurred  by  a  prudent  person  under  the  circumstances  prevailing  at  the  time  the  decision  was  made  to  incur  the 
cost. 
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DSF  Provides  a  Uniform  Approach  for  Designing  and  Evaluating  Stability  Operations 
Programs 

After  the  task  order  award,  OTI  determined  that  it  wanted  DAI  to  implement  the  ASI-East  program  under 
a  new  methodology  that  departed  from  the  relatively  rudimentary  Tactical  Conflict  Assessment  and 
Planning  Framework  tool  and  moved  toward  a  more  rigorous  programming  and  monitoring  and 
evaluation  methodology.  OTI’s  DSF  methodology  did  not  follow  the  “burn-rate”  approach  to  program 
spending,  which  emphasizes  spending  funds  as  quickly  as  feasible.  In  contrast,  the  DSF  programming 
model  takes  a  more  strategic  view  toward  spending,  which  DATs  Chief  of  Party  described  as  a  “small  and 
smart”  approach  to  stabilization  programming.  Both  OTI  and  DAI  agreed  on  the  need  for  this  new 
programming  methodology  and,  in  coordination  with  COIN  Training  Academy,  developed  the  DSF  in 
early  2010  under  the  ASI-East  program  as  an  integrated  civilian-military  methodology  for  identifying, 
implementing,  and  measuring  the  impact  of  stabilization  activities. 

DSF  provides  civilians,  military,  and  GIRoA  officials  with  a  shared  terminology  and  a  unified  assessment 
process  for  gauging  the  stability  picture  across  a  district  or  area,  identifying  local  sources  of  instability, 
and  developing  coherent  and  tailored  programs  and  interventions  to  address  them.  DSF  emphasizes 
assessing  results  and  modifying  and  adapting  program  activity  based  on  empirical  data.  The  DSF  system 
requires  a  three-tier  monitoring  and  evaluation  system,  which  measures  output,  impact,  and  overall 
stability  conditions  at  the  district  level.  DSF  implementation  efforts  have  included  developing  several 
iterations  of  the  methodology  and  training  more  than  1,200  civilian  and  military  personnel. 

Figure  1  shows  the  basic  elements  of  the  DSF  methodology  and  the  four-phase  process,  which  starts  with 
situational  awareness  and  concludes  with  monitoring  and  evaluation  efforts  with  a  feedback  loop  back  to 
the  first  phase  in  the  process.  The  DSF  is  implemented  at  the  district-level  by  a  civilian  and  military 
working  group  charged  with  designing  and  implementing  a  set  of  projects  to  address  specific  sources  of 
instability  that  are  aligned  with  stabilization  programming  goals. 


Figure  1:  DSF  System  Flowchart 
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Grant  Activity  and  Funding 

ASI-East  program  resources  are  largely  distributed  through  a  “grants  under  contract”  approach,  which 
relies  on  DAI  as  the  contractor  to  award,  administer,  assess,  and  close-out  grants  to  local  community 
implementers.  As  of  October  16,  201 1,  DAI  awarded  461  grants  at  a  total  cost  of  about  $21.2  million  in 
ten  districts  distributed  across  Nangarhar,  Kunar,  Wardak,  and  Paktika  provinces  in  eastern  Afghanistan. 
These  grants  cover  a  wide-range  of  projects,  such  as  improvements  in  agriculture  and  irrigation; 
assessments  of  community  and  district  availability  of  services;  media  outreach  efforts  that  showcase 
GIRoA’s  involvement  in  providing  needed  services;  and  a  variety  of  infrastructure  projects,  including 
repairing  and  building  new  roads,  retaining  walls,  flood  ditches,  and  sanitation  facilities.  Figure  2 
provides  a  breakout  of  ASI-East  activity  funding  and  the  number  of  grants  awarded  through  October 
2011. 


Note:  Shura/jirga  is  an  Afghan  meeting  of  a  decision  making  body  (typically  tribal/village  elders).  In  the  context  of  the  ASI- 
East  program,  these  meetings  were  convened  to  discuss  grant-funded  activities.  Scribe  activities  were  used  to  gather  and 
record  certain  demographic  information. 


PROGRAM  SPENDING  MARKED  BY  HIGH  OPERATING  EXPENSES  AND 
CERTAIN  QUESTIONABLE  COSTS 

While  DAI  did  not  exceed  the  $151  million  cost  ceiling  for  the  ASI-East  task  order,  program  expenditures 
largely  covered  operating  expenses,  as  opposed  to  activity  level  spending.  DAI  officials  noted  that 
relatively  low  activity  spending  levels  represented  a  “small  and  smart”  approach  to  stabilization,  which 
OTI  officials  endorsed  based  on  the  application  of  the  DSF  methodology  to  ASI-East  programming 
decisions.  SIGAR  identified  over  $590,000  in  questionable  program  costs  that  will  need  to  be  reviewed 
by  USAID  to  determine  whether  these  costs  were  allowable,  allocable,  and  reasonable.  Finally,  SIGAR 
noted  a  limited  number  of  procedural  and  compliance  deficiencies  associated  with  two  of  DATs  key 
business  systems. 
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ASI-East  Implementation  Marked  by  Low  Activity  Spending  and  High  Operating  Costs 

SIGAR  found  that  DAI  had  spent  only  25  percent  of  expended  funds  on  project  activities  through  October 
2011,  with  the  balance  of  funds  spent  on  operating  expenses.  This  spending  pattern  was  not  consistent 
with  the  implied  spending  ratios  in  the  task  order,  as  stipulated  at  the  time  of  award.  Specifically,  the  task 
order  stipulated  that  up  to  $83  million  could  be  spent  on  program  activities  and  up  to  $68  million  could  be 
spent  on  operating  costs.4  Therefore,  the  implied  spending  ratio  was  55  percent  on  program  activities  and 
45  percent  on  operating  costs. 

DAI  representatives  noted  that  the  ASI-East  program  has  been  implemented  under  a  “small  and  smart” 
approach  to  stabilization  based  on  the  application  of  the  DSF  methodology.  One  result  of  the  application 
of  this  “small  and  smart”  approach  to  programming  is  that  the  ratio  of  project  costs  to  operating  costs  is 
not  in  line  with  the  implied  spending  ratio  in  the  task  order.  DAI  officials  noted  that  the  “small  and 
smart”  approach  to  programming  was  approved  by  OTI  officials  because  it  represents  the  logical  outcome 
of  applying  the  DSF  to  programming  decisions.  DAI  officials  also  noted  that  this  approach  contrasts  with 
the  “burn  rate”  approach  typically  used  by  other  assistance  programs.  They  also  noted  that  unlike  many 
other  USAID  programs  which  often  involve  large  capital  expenditures  for  infrastructure  development, 
ASI-East’ s  use  of  small  grants  to  local  implementers  tended  to  depress  program  spending  levels. 

DAI  officials  noted  that  administering  and  overseeing  these  small  grants  requires  the  same  level  of  effort 
and  operating  expenses  as  administering  and  overseeing  large  dollar  contracts  and  assistance  instruments. 
DAI  also  stated  the  required  operating  expenditures  needed  to  establish  and  operate  the  ASI-East  program 
were  not  excessive  or  unreasonable.  Specific  operating  costs  included  office  and  living  space  rental  in 
Kabul  and  locations  across  Afghanistan,  security  costs,  and  the  large  number  of  personnel  needed  to 
implement  and  oversee  the  hundreds  of  grants  awarded  under  the  program. 

DAI  Charged  More  Than  $590,000  in  Questionable  Costs  to  the  ASI-East  Program 

We  generally  found  that  the  ASI-East  incurred  costs  were  allowable,  allocable,  and  reasonable  with  the 
following  exceptions: 

•  OTI  originally  contracted  directly  with  a  contractor  (QED)  to  provide  independent  monitoring 
and  evaluation  services  for  the  ASI-East  program.  QED  subcontracted  with  Altai  Consulting  to 
provide  these  services.  The  contract  between  OTI  and  QED  expired  in  July  201 1.  To  provide 
follow-on  monitoring  and  evaluation  services,  OTI  requested  that  DAI  subcontract  with  Altai 
Consulting  to  perform  additional  cycles  of  monitoring  and  evaluation  in  three  districts,  at  a  total 
cost  of  over  $500,000.  OTI  staff  developed  a  conflict  of  interest  mitigation  plan  to  address 
questions  of  independence  and  oversight  raised  by  this  arrangement.  The  plan  called  for  OTI  to 
provide  oversight  of  the  subcontract  and  directly  receive  all  subcontract  deliverables.  DATs  role 
in  administering  the  subcontract  was  essentially  limited  to  reviewing  and  approving  invoices  for 
payment.  In  response  to  the  draft  of  this  report,  OTI’s  Contracting  Officer  provided  additional 
information  regarding  the  basis  for  making  a  determination  that  the  over  $500,000  was  allowable. 
The  response,  however,  did  not  address  the  subcontractor  oversight.  The  mitigation  plan  that  was 
implemented  by  OTI  between  DAI  and  Altai  limited  DATs  ability  to  provide  adequate 
subcontractor  oversight,  as  required  by  FAR  42.202(e)(2)  .  DATs  role  in  administering  the 
subcontract  was  essentially  limited  to  reviewing  and  approving  invoices  for  payment.  Therefore, 
we  continue  to  question  whether  the  cost  of  over  $500,000  was  allowable. 

•  We  found  that  the  $3,400  cost  of  a  household  effects  demobilization  shipment  that  was  related  to 
a  DAI  Iraq  project  was  incorrectly  charged  to  the  ASI-East  program.  Per  the  FAR,  such  costs  are 
unallocable. 


4 


Dollar  figures  based  on  a  revised  budget  approved  under  modification  4,  approved  July  27,  2010,  to  the  task  order. 
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•  DAI  maintained  full  comprehensive  and  war  insurance  on  purchased  armored  security  vehicle 
while  they  were  in  the  United  States  waiting  for  Afghan  customs  approval  for  importation. 
Property  damage  insurance  alone  likely  would  have  provided  sufficient  coverage  and  resulted  in 
an  estimated  cost  savings  of  $50,000  if  DAI  had  gone  with  this  lower  cost  option.  These 
additional  costs  appear  to  be  unreasonable  and  therefore  unallowable  per  FAR  requirements. 

•  While  waiting  for  its  purchased  armored  vehicles  to  arrive  in  Afghanistan,  DAI  leased  armored 
vehicles,  which  were  covered  with  comprehensive  and  war  insurance  per  the  terms  of  the  rental 
agreement.  During  that  time,  DAI  traded  in  old  leased  vehicles  for  newer  ones.  When  this 
occurred,  DAI  added  the  new  vehicles  to  its  insurance  policy;  however,  it  did  not  remove  the 
leased  armored  vehicles  no  longer  being  used.  As  a  result,  DAI  billed  $38,500  in  201 1  for 
insurance  costs  on  vehicles  that  were  not  being  utilized  by  the  project;  these  additional  costs 
appear  to  be  unreasonable  and  therefore  unallowable  per  FAR  requirements. 

Relatively  Minor  Deficiencies  in  DAPs  Time  Keeping  and  Billing  Related  Processes  Noted 

We  reviewed,  analyzed,  and  performed  limited  testing  on  controls  of  DATs  time  keeping  and  billing 
systems.  We  followed  the  General  Accounting  Office  Standards  for  Internal  Control  in  the  Federal 
Government ,  which  states  that  physical  controls  must  be  in  place  to  secure  and  safeguard  vulnerable 
assets  at  risk  of  loss  or  unauthorized  use.5  Generally,  DATs  time  keeping  and  billing  policies  and 
procedures  functioned  properly,  with  a  few  minor  exceptions: 

•  With  regard  to  DAI  Afghan  staff  timesheets,  we  observed  multiple  occurrences  where  individuals 
entered  time  for  other  individuals  on  physical  timesheets  displayed  in  the  lobby  of  DATs 
headquarters  in  Kabul. 

•  DAI  supervisors  sign  Afghan  timesheets  on  a  monthly  basis,  as  opposed  to  the  bi-weekly  reviews 
conducted  of  expatriate  staff  timesheets. 

•  We  noted  several  instances  where  DAI  did  not  initially  furnish  complete  supporting 
documentation  for  paid  invoices.  For  example,  travel  invoices  did  not  have  the  proper  contracting 
officer  approval  letter  attached,  procurement  invoices  did  not  have  all  bidding  documentation 
attached,  and  a  brokerage  fee  had  no  attached  invoice.  DAI  was  able  to  furnish  all  supporting 
documentation  once  requested.  DAI  noted  that  while  it  does  not  always  keep  all  supporting 
documentation  attached  to  invoices,  supporting  documentation  is  available  in  a  separate  location. 
SIGAR  questions  this  arrangement,  which  does  not  provide  adequate  assurance  that  OTI 
contracting  staff  have  sufficient  supporting  information  available  when  reviewing  and  approving 
invoice  claims. 

•  When  leasing  items,  contractors  generally  perform  a  buy  versus  lease  cost  analysis  to  ensure  that 
the  government  receives  the  best  value  for  the  money.  This  cost  analysis  is  not  only  used  for 
contractor  purchases,  but  also  for  subcontracter  purchases.  DATs  security  subcontractor, 
Edinburgh  International,  leased  several  items  including  computers  and  printers,  body  armor,  night 
vision  goggles,  and  soft  skin  vehicles  without  conducting  a  buy  versus  lease  analysis.  In  certain 
cases  it  would  have  been  cheaper  to  procure  leased  items.  For  example,  SIGAR  estimates  that 
$20,000  could  have  been  saved  by  purchasing  computers  and  printers  directly  compared  to 
leasing  the  same  equipment.  According  to  DAI,  Edinburgh  International  did  not  conduct  buy 
versus  lease  cost  analyses  for  the  items  discussed  above. 


5U.S.  General  Accounting  Office,  Standards  for  Internal  Control  in  the  Federal  Government ,  GAO-AIMD-00- 
21.3.1  (Washington,  D.C.:  Nov.  1999).  See  also  Development  Alternatives,  Inc.:  2010  Financial  Statement  and 
Audit  Results  (Ernst&Young,  May  5,  201 1).  DATs  external  auditors’  opinions  of  its  financial  statements  for  2009 
and  2010  considered  DATs  internal  controls  when  performing  the  financial  audits. 
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PROGRAM  OVERSIGHT,  MONITORING,  AND  EVALUATION  SYSTEMS 
IMPLEMENTED  BUT  CERTAIN  ADMINISTRATIVE  REQUIREMENTS  NOT  MET 


DAI  and  OTI  implemented  program  oversight  and  monitoring  and  evaluation  systems  through  a  variety  of 
means,  including  a  “grants  under  contract”  tool  managed  by  DAI  that  was  adequately  implemented; 
however,  certain  administrative  requirements  of  the  ASI-East  task  order  related  to  project  oversight  were 
not  fully  met.6  Program  monitoring  and  evaluation  centered  on  the  three-tier  evaluation  system  called  for 
by  the  DSF  methodology.  Developing  this  three-tier  monitoring  and  evaluation  system  took  16  months 
and  final  output,  impact,  and  overall  stability  results  are  not  expected  until  sometime  in  2012. 

Preliminary  results,  however,  show  significant  and  positive  results  at  the  project  level  while  instability 
remains  a  problem  across  ASI-East’s  ten  program  districts  in  general.  USAID’s  efforts  to  evaluate  final 
ASI-East’ s  program  results  will  be  critical  to  informing  short  and  long-term  decisions  regarding  whether 
and  how  extensively  this  new  methodology  should  be  implemented  in  future  stabilization  efforts. 

Although  we  noted  significant  levels  of  program  oversight,  monitoring,  and  evaluation,  we  also  identified 
a  limited  number  of  administrative  issues  relating  to  program  oversight. 

DAI  and  OTI  Provided  Program  Oversight 

DAI  and  OTI  staff  provided  oversight  of  ASI-East  program  activities  in  ten  districts  across  the  Regional 
Command  East  platform  through  a  variety  of  means.  One  key  component  of  provided  oversight  is  the 
“grants  under  contract”  tool  that  is  managed  by  25  DAI  expatriate  staff  overseeing  244  DAI  Afghan 
employees  based  in  Kabul,  Jalalabad,  and  ten  District  Development  Teams.  This  process  requires  that  a 
series  of  award,  implementation,  and  close-out  procedures  be  followed  for  each  grant.  Specific 
requirements  include  a  document  checklist,  which  identifies  required  grant  documents,  procurement 
documents,  financial  documents,  and  reporting  and  evaluation  documents.  In  addition,  financial  action 
status,  OTI  database  inputs,  and  approval  signatures  from  DAI  and  OTI  representatives  are  required  for 
each  grant.  To  assess  whether  DAI  and  OTI  exercised  adequate  controls  and  oversight,  we  reviewed  a 
judgmental  sample  of  20  completed  and  closed  grants  awarded  by  DAI,  as  of  December  2011,  and  found 
controls  to  be  adequate  for  each  activity.  Specifically,  we  found  that  all  activity  folders  contained  the 
information  required  by  DAI  for  documenting  the  grant  award,  implementation,  and  close-out  process. 

Located  in  Washington,  D.C.,  OTI’s  COTR  is  responsible  for  overseeing  the  ASI-East  program  and 
approving  the  payment  of  all  project  invoices.  The  COTR’s  oversight  efforts  are  supplemented  by  an 
extensive  network  of  OTI  staff  in  Afghanistan,  which  includes  a  Country  Representative  based  at  the 
embassy  in  Kabul  and  a  Field  Operations  and  Project  Manager  and  Deputy  Regional  Representative 
located  in  Jalalabad.7  These  individuals  provide  additional  oversight  by  working  directly  with  DAI  staff 
in  Kabul  and  Jalalabad  and  with  Regional  Command  East  representatives.  All  OTI  staff  have  access  to 
DAI -prepared  weekly,  annual,  and  quarterly  performance  reports,  which  typically  include  program 
description,  country  situation,  program  activity  highlights,  successes  and  challenges,  activity  summary, 
program  appraisal,  and  DSF  activity  status. 

OTI  also  provided  oversight  of  the  ASI-East  program  by  conducting  periodic  program  performance 
reviews  and  strategic  review  sessions.  These  reviews  are  designed  to  provide  a  review  of  task  order 
progress  and  key  challenges  by  staff  not  directly  associated  with  the  program.  OTI  conducted  program 
performance  reviews  of  ASI-East’s  activities  in  November  2010  and  October  2011.  Key  findings  from 
these  two  reviews  include  the  following: 


6The  ASI-East  task  order  outlines  the  need  for  DAI  in  coordination  with  USAID/OTI  to  institute  a  (1)  a  monitoring 
and  evaluation  system;  (2)  weekly,  quarterly,  and  annual  performance  reports;  (3)  participation  in  strategic  review 
sessions;  and  (4)  the  maintenance  of  the  OTI  activity  database  as  the  backbone  of  the  performance  management 
system. 

7As  of  October  2011,  seven  OTI  staff  members  were  located  in  Kabul,  two  were  located  at  the  regional  command- 
East  platform  in  Bagram  Air  Force  base,  and  six  others  were  stationed  at  other  Coalition  Forces  forward  operating 
bases  in  the  districts  in  which  OTI  works. 
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•  The  USAID  Kabul  Mission  should  maintain  stabilization  programming  capacity,  especially  in  the 
East  and  South  regions,  throughout  transition  over  the  next  3  years,  despite  engagement 
constraints. 

•  OTI  and  DAI  need  to  prepare  exit  criteria  and  transition  strategies  to  help  move  districts  from  the 
“hold”  to  the  “build”  phase  of  the  COIN  strategy. 

•  OTI  should  develop  a  new  task  order  under  the  SWIFT  III  IQC  with  a  single  implementing 
partner  to  consolidate  ASI-East  and  ASI-South.8 

To  review  and  adjust  program  activities,  OTI  and  DAI  also  conducted  bi-annual  strategic  review  sessions. 
The  most  recent  session  was  conducted  in  Kabul  in  October  2011  and  was  widely  attended  by  OTI  and 
DAI  staff  from  Washington,  D.C.,  and  Kabul  and  Jalalabad,  as  well  as  district  field  personnel. 

Participants  discussed  program  strategy  and  monitoring  and  evaluation  efforts,  including  DATs  first 
“Lessons  Learned”  report.9  This  report  analyzed  90  program  activities  that  were  closed  and  completed  in 
order  to  identify  best  practices  to  apply  to  future  activities. 

Certain  Program  Oversight  Elements  Not  Fully  Implemented 

SIGAR  found  that  certain  administrative  and  management  requirements  relating  to  program  oversight  had 
not  been  fully  met,  as  shown  in  table  1 . 


Table  1:  Administrative  and  Management  Issues  Relating  to  Program  Oversight 


Issue 

Description 

Data  validity  concerns 

The  IQC  and  task  order  state  that  DAI  should  maintain  and  use  the  USAID/OTI  Activity 
Database  as  the  “backbone”  to  generate,  develop,  manage,  monitor,  evaluate,  and  report 
on  program  activities.  This  database  has  several  fields  that  need  to  be  populated  with 
information  that  captures  the  status  of  the  program  activity.  Per  an  understanding  with  OTI 
officials,  DAI  used  its  Technical  Administrative  Management  Information  System  (TAMIS) 
to  track  program  activities.  Information  from  this  system  is  being  modified  on  a  weekly 
basis  in  order  to  provide  a  “data  dump”  into  the  USAID/OTI  Activity  Database.  We  found 
that  OTI  accepts  the  data  from  DAI’s  TAMIS  system  without  conducting  periodic  data 
reliability  checks.  OTI  officials  noted  that  they  monitor  the  data  uploaded  into  the  OTI 
Activity  Database;  however,  we  were  told  that  this  monitoring  is  performed  in  Washington, 
D.C.  and  not  locally  in  Kabul,  where  better  checks  on  data  reliability  could  be  performed. 
SIGAR  reviewed  data  for  similar  periods  in  the  OTI  Activity  Database  and  TAMIS  and 
found  discrepancies  that  overstated  TAMIS  reported  activity.  While  these  discrepancies 
were  ultimately  explained  by  OTI  officials,  we  believe  data  validation  should  occur  at  the 
field  level  to  ensure  consistent  reporting  in  both  systems. 

Fraud  reporting 

We  noted  three  instances  of  fraud  not  reported  to  USAID  officials  in  a  timely  manner  as 
required  by  USAID  policy  and  FAR  11,12.  Fraud-related  costs,  per  DAI’s  disclosed 
practices  and  USAID  guidance,  are  to  be  set  aside  in  an  unallowable  account  so  that  these 
costs  are  not  billed  to  and  reimbursed  by  the  U.S.  government.  We  noted  that  DAI 
identified  three  instances  of  fraud  between  December  12,  2010,  and  April  19,  2011,  that 
totaled  approximately  $1 ,000.  In  two  instances,  the  fraud  was  discovered  before  the 
expense  was  reimbursed  to  the  employee  and  captured  in  DAI’s  financial  software.  For 
the  final  instance,  the  fraudulent  amount  was  processed  and  captured  in  DAI’s  financial 
software.  After  the  fraudulent  amount  was  discovered,  DAI  asserts  that  the  amount  was 
charged  to  a  non-billable  account  and  was  not  billed  to  USAID.  However,  we  have  not 
been  able  to  confirm  whether  the  unallowable  cost  was  actually  charged  to  a  non-billable 
account.  DAI  has  since  changed  its  policy  with  regards  to  reporting  fraud  based  on 

SIGAR’s  observations.  DAI  will  now  report  all  instances  of  fraud,  regardless  of  amount  or 
if  it  was  billed  to  USAID,  to  comply  with  USAID  policies  and  FAR  requirements. 

g 

In  February  2012,  OTI  awarded  a  task  order  to  consolidate  East  and  South  program  activity  under  a  single  task 
order  to  Creative  Associates  International. 

9DAI  formally  transmitted  these  lessons  learned  in  its  report:  DAI,  Monitoring  &  Evaluating  ASI-East:  Key  Results 
from  Output  Monitoring  and  Overall  Stability  Monitoring ,  October  23,  2011. 
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Issue  Description 

Subcontract  oversight  DAI’s  oversight  of  its  subcontract  with  its  security  contractor’s,  Edinburgh  International,  is 
limited.  The  Defense  Contract  Audit  Agency’s  audit  manual  notes  that  prime  contractor 
oversight  of  subcontractors  should  1)  include  subcontract  award,  (2)  include  technical  and 
financial  performance  monitoring,  (3)  ensure  that  indirect  rate  proposals  and  annual  rate 
adjustments  are  submitted  on  a  timely  basis  (incurred  cost  proposal),  and  (4)  ensure  that 
payment  to  the  subcontractor  for  the  work  accomplished  was  in  accordance  with  the 
subcontract  terms  and  based  on  allowability,  allocability  and  reasonableness  principles. 
DAI  did  not  fully  follow  this  guidance  based  on  our  review  of  documentation  and  interviews 
with  DAI  staff.  DAI  disputes  this  characterization,  noting  that  requirements  one  and  two 
have  been  met  through  various  contracting  and  security  officials  meeting  with  Edinburgh 
International  officials  and  reviewing  Edinburgh  International  contract  requirements  and 
billings.  DAI  asserts  that  it  has  addressed  the  third  oversight  requirement  by  negotiating 
Edinburgh  International’s  indirect  rate  at  the  beginning  of  the  contract  and  will  adjust  the 
indirect  rate  at  the  end  of  the  contract.  DAI  asserts  that  Edinburgh  International  has  an 
adequate  billing  system  that  ensures  costs  billed  are  allowable,  allocable  and  reasonable. 
Additionally,  DAI  reviews  Edinburgh  International’s  billings  to  ensure  their  accuracy. 
Despite  these  cited  instances  of  provided  oversight,  we  found  that  DAI  is  not  providing 
adequate  oversight  as  required  by  the  task  order  because  we  found  no  evidence  that  it  is 
performing  yearly  reviews  of  Edinburgh  International’s  overhead  rates  to  determine  their 
reasonableness.  In  addition,  DAI  has  not  been  able  to  provide  evidence  that  it  reviewed 
Edinburgh  International’s  key  business  systems  to  ensure  that  process  and  internal 
controls  are  adequate  for  determining  that  the  costs  Edinburgh  International  is  incurring 
and  billing  are  allowable,  allocable,  and  reasonable. 

Source:  SIGAR  analysis. 


Monitoring  and  Evaluation  Systems  Provide  Initial  Insights  on  Program  Results  and  DSF 
Implementation 

Beginning  with  the  April  2010,  arrival  of  its  current  monitoring  and  evaluation  director,  DAI  sought  to 
develop  a  three-tier  monitoring  and  evaluation  system  based  on  DSF  requirements  to  assess,  among  other 
items,  ASI’s  effect  on  the  overarching  strategic  goal  of  bolstering  the  capacity  and  legitimacy  of  the 
Afghan  government  at  the  community  level.  OTI  contracted  separately  with  a  third-party  provider,  QED, 
which  then  subcontracted  with  Altai  Consulting,  in  June  2010,  for  an  independent  assessment  of  program 
outcomes  and  impact  using  a  different  assessment  methodology.  Preliminary  monitoring  and  evaluation 
results  as  of  December  2011  from  both  efforts  show  a  range  of  documented  outputs  and  encouraging 
impact  at  the  program  level,  while  overall  stability  was  judged  as  being  low  across  ASI-East’s  ten 
programming  districts.  When  DAI  and  Altai  Consulting’s  evaluation  efforts  are  completed  later  in  2012, 
OTI  and  DAI  believe  they  will  have  the  information  they  need  to  evaluate  ASI’s  overall  impact  and  the 
relative  success  of  DSF  as  a  stability  programming  model.  Appendix  II  contains  further  details  on  the 
DSF  and  Altai  Consulting  monitoring  and  evaluation  methodologies  and  the  preliminary  results  of  these 
efforts. 

DSF  and  Altai  Consulting  Monitoring  and  Evaluation  Methodologies 

The  DSF  monitoring  and  evaluation  methodology  focuses  on  measuring  results  at  three  levels:  output, 
impact,  and  overall  stability.10  DATs  monitoring  and  evaluation  unit  designed  and  implemented  a  system 
that  follows  the  DSF  methodology.  DATs  monitoring  and  evaluation  staff  collected  and  analyzed  data 
for  program  output  and  impact  measurement  and  contracted  with  RSI  Consulting  to  conduct  data 


10Evaluation  activities  normally  focus  on  outputs,  outcomes,  and  impacts.  As  noted  in  our  report,  the  DSF 
evaluation  methodology  does  not  currently  address  outcomes.  Outputs  are  defined  as  measurable  outputs  from 
program  activities  such  as  miles  of  drainage  ditches  built  or  hours  of  media  broadcast  each  week.  Outcomes  are 
defined  as  the  observed  effects  of  the  outputs  on  the  beneficiaries  of  the  program  such  as  improved  agriculture 
yields  or  improved  perceptions  of  the  Afghan  government.  Impact  is  defined  as  the  degree  to  which  the  observed 
outcome(s)  can  be  attributed  to  the  funded  activities. 
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collection  and  analysis  efforts  for  measuring  ASI’s  contributions  to  overall  stability.  DAI  data  collection 
efforts  for  all  three  levels  will  be  concluded  before  the  task  order  ends  in  September  2012. 

Issues  and  questions  addressed  by  DAI  using  the  DSF  model  at  each  level  include  the  following: 

•  Level  1/output:  Have  scheduled  activities  been  completed  and  successfully  implemented?  Are 
external  factors  affecting  the  implementation  of  scheduled  activites? 

•  Level  2/impact:  Is  the  intended  impact  or  change  being  seen  in  the  environment?  Do  observed 
changes  in  the  environment  represent  progress  toward  stated  objectives  and  a  diminishment  of  the 
root  causes  of  instability?  How  are  external  factors  influencing  or  causing  any  observed  changes? 

•  Level  3/overall  stability:  Are  the  collective  efforts  (ASI-East  and  any  other  programming  efforts) 
to  address  the  sources  of  instability  in  a  given  district  leading  to  improved  stability  as  measured 
by  a  standard  set  of  seven  indicators?  Does  trend  data  over  time  indicate  that  the  security 
situation  is  getting  better  or  deteriorating  in  a  given  district? 

Altai  Consulting  monitoring  and  evaluation  consists  of  four  cycles  that  are  scheduled  to  conclude  in  June 
2012.  The  first  cycle  focused  on  monitoring  program  outputs.  Cycles  2  through  4  will  address 
monitoring,  impact,  and  overall  stability  questions  in  two  districts  for  cycle  2  and  three  districts  for  cycles 
3  and  4. 

Reports  Show  Mixed  Results  on  Output  and  Overall  Stability 

DAI  issued  its  first  formal  report  on  level  1/output  results  in  September  201 1. 11  Issued  as  a  “lessons 
learned”  report,  the  document  presents  DATs  analysis  of  90  completed  program  activities  which  found 
that  these  activities  had  achieved  their  intended  purposes.  The  report  also  highlighted  success  stories  that 
should  help  guide  future  programming  decisions.  The  report  focused  on  how  to  improve  (1)  activity 
implementation  through  community  buy-in,  physical  presence  of  GIRoA  officials,  and  publicizing 
projects;  and  (2)  activity  operations  through  adopting  the  Afghan  First  Initiative,12  vetting  suppliers  and 
trainers,  providing  performance  feedback,  and  ensuring  that  cash  payments  for  work  are  timely. 

In  October  2011,  RSI  Consulting  provided  DAI  with  a  final  report  on  level  3/overall  stability  results 
demonstrating  that  overall  stability  in  all  ten  ASI-East  districts  is  relatively  low  based  on  the  key 
indicators  of  stability.13  Specifically,  the  low  levels  of  stability  are  attributed  to  security  issues  within 
districts,  a  lack  of  confidence  in  the  local  and  national  government,  and  underlying  cultural  differences 
between  locals  and  outsiders. 

Altai  Consulting  independent  monitoring  and  evaluation  efforts  have  generally  confirmed  the  preliminary 
results  revealed  by  DATs  early  monitoring  and  evaluation  efforts. 

DATs  First  Impact  Assessment  Released  in  January  2012 

In  January  2012,  DAI  released  its  first  assessment  of  the  impact  ASI-East  programming  had  on  nine 
sources  of  instability  in  four  districts.14  The  goals  of  the  assessment  were  to  identify  stability-related 
changes  in  the  selected  districts  over  the  last  18  months,  to  examine  the  factors  that  explain  those 
changes,  and  to  understand  the  effectiveness  of  different  programming  strategies  used  in  each  district. 

nDAI,  Lessons  Learned  Report:  Afghanistan  Stabilization  Initiative-East ,  September  19,  201 1. 

12 

Since  March  2006,  U.S.  military  and  civilian  agencies  operating  in  Afghanistan  have  taken  steps  to  ensure  that  a 
greater  number  of  contracts  are  awarded  to  Afghan  companies.  Over  time,  these  efforts  have  become  collectively 
known  as  the  Afghan  First  Initiative. 

13 

RSI  Consulting,  Overall  Stability  Assessment  in  10  Districts  in  Afghanistan ,  October  2011. 

14USAID  published  DATs  assessment  in  the  following  document,  USAID,  Preliminary  Impact  Assessment  in  Four 
Districts  in  Eastern  Afghanistan:  Khogyani,  Marawara,  Saydabad,  and  Ur  gun,  January  16,  2012. 
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This  assessment  concluded  that  ASI-East  programming  was  found  to  have  a  positive  impact  on  5  of  the 
9  sources  of  instability.  In  one  case,  ASI-East  programming  was  found  to  have  reduced  the  source  of 
instability.  In  three  cases,  the  data  gathered  could  not  establish  a  positive  or  negative  effect.  The  report 
noted  that  because  data  was  collected  in  the  middle  of  programming,  results  of  this  assessment  are  not 
comprehensive  or  predictive  of  the  final  impact  of  ASI-East.  Although  preliminary,  these  findings  offer 
feedback  for  ASI-East  as  it  refines  its  strategy  in  the  final  phase  of  programming  and  for  follow-on 
application  of  the  DSF  methodology.15 

The  assessment  concluded  that  DSF  is  an  effective  and  versatile  methodology  that  allowed  it  to  identify 
changes  in  the  environment  and  adjust  ASI-East  programming  decisions  accordingly.  The  assessment 
also  provided  recommendations  for  improved  implementation  of  the  DSF  including  (1)  beginning 
program  activities  after  an  area  has  reached  the  “clear”  stage  of  the  COIN  strategy,  (2)  increasing 
coordination  with  the  military  is  needed  so  programming  can  start  immediately  after  military  operations 
cease,  (3)  increasing  community  involvement  by  GIRoA,  and  (4)  performing  community  assessments 
before  programming  begins. 

Project  Outcomes  Not  Included  in  DSF  Monitoring  and  Evaluation  Methodology 

We  did  not  identify  methodological  flaws  or  problems  with  how  DAI  and  Altai  Consulting  are 
implementing  their  monitoring  and  evaluation  efforts.  However,  we  did  note  that  the  DSF  methodology 
does  not  include  measuring  outcomes.  Measuring  project  outcomes  would  help  implementers  gain 
feedback  on  how  projects  are  received  by  the  beneficiaries  and  provide  a  leading  indicator  for  level  2 
impact  analysis.  Adding  project  outcomes  to  level  1  analysis  on  outputs  will  allow  for  earlier  feedback 
and  project  adjustment  to  obtain  better  results. 

We  also  noted  that  level  3  analysis  on  overall  stability  was  first  reported  28  months  into  the  task  order. 
While  the  analysis  provided  valuable  information,  it  would  have  been  better  if  it  had  been  performed  at 
the  beginning  of  the  task  order  or  when  stabilization  efforts  begin  in  new  districts.  This  would  have 
provided  baseline  information  for  subsequent  results  analysis.  This  is  a  key  lesson  learned  for  all  future 
stabilization  programs  using  the  DSF  methodology. 


OTI  LACKS  APPROVED  DISENGAGEMENT  CRITERIA  AND  AN  OVERALL 
PROGRAM  EXIT  STRATEGY  TO  MOVE  DISTRICTS  FROM  THE  HOLD  TO  BUILD 
PHASE  OF  THE  COIN  STRATEGY 

Despite  nearly  3  years  of  program  activity,  none  of  DATs  target  districts  have  transitioned  to  the  “build” 
phase  of  the  COIN  strategy. 16  SIGAR  found  that  approved  disengagement  criteria  for  individual  districts 
and  an  overall  transition  strategy  for  the  program  remain  to  be  developed.  OTI  Afghanistan  has  drafted  a 
set  of  disengagement  criteria  for  districts  that  could  provide  a  clear  basis  for  deciding  to  terminate  “hold” 
programming  in  a  given  district;  however,  the  draft  criteria  lack  any  target  indicators  or  benchmarks  to 
provide  the  basis  for  such  a  decision.  These  target  indicators  or  benchmarks  would  likely  be  tied  to  the 
output  and  impact  measures  which  DAI  and  OTI’s  monitoring  and  evaluation  contractor  are  pursuing. 
Under  the  provisions  of  the  SWIFT  III  IQC,  OTI  is  to  develop  an  exit  strategy  for  each  OTI  program 


15The  report  noted  other  methodology  limitations.  The  most  significant  limitation  was  the  lack  of  baseline 
information,  which  made  it  difficult  to  compare  the  situation  before  the  ASI-East  program  began  addressing  the 
sources  of  instability.  In  addition,  the  assessment  was  conducted  during  the  summer,  a  period  when  violence 
typically  increases,  which  may  have  influenced  respondents’  assessments  of  stability  in  their  districts.  Finally,  the 
ASI-East  program  was  only  one  of  several  stabilization  programs  in  the  operational  districts,  other  actors  may 
influence  the  stability  objectives  ASI-East  is  designed  to  achieve.  This  presented  a  challenge  to  attributing  observed 
changes  to  the  ASI-East  program. 

16In  responding  to  our  draft  report,  USAID  noted  that  districts  in  Wardak  and  Kunar  provinces,  where  ASI-East 
programming  took  place,  were  recently  identified  as  ready  for  transfer  to  GIRoA  (see  app.  III). 
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within  1 8  months  of  the  program  start-up,  including  the  ASI-East  program.  OTI  has  not  yet  developed  an 
exit  strategy  to  guide  its  programming  efforts,  which  must  be  integrated  with  U.S.  government  and 
mission-wide  transition  planning  efforts  for  Afghanistan.  However,  OTI  officials  noted  that  such  a 
transition  strategy  would  be  developed  in  connection  with  the  3 -year  extension  of  the  ASI  program  that 
began  in  February  2012. 

OTI  Lacks  Approved  Disengagement  Criteria  and  a  Program  Exit  Strategy 

OTI  programs  are  designed  to  be  short,  targeted,  and  flexible.  In  accordance  with  its  SWIFT  III  IQC, 

OTI  generally  plans  to  exit  a  country  program  within  2  years  of  initiating  activity.  In  Afghanistan,  OTI 
expected  that  the  3  year  ASI  effort  would  lead  to  at  least  some  districts  transitioning  from  the  “hold”  to 
the  “build”  phase  of  the  COIN  strategy.  All  OTI  programs  are  required  to  have  an  exit  strategy  developed 
within  18  months  of  program  start-up.  Both  OTI  led  program  performance  reviews  of  ASI-East  program 
noted  the  need  for  OTI  to  prepare  exit  criteria  and  exit  strategies  to  help  move  districts  from  the  “hold”  to 
the  “build”  phase  of  the  COIN  strategy.  This  issue  was  also  identified  as  an  area  of  concern  by  the 
USAID  Office  of  the  Inspector  General  in  its  November  2011  report  on  the  implementation  of  the  ASI 
program  in  southern  Afghanistan. 17  The  report  noted  that  the  mission  does  not  have  a  transition  plan  and 
recommended  that  USAID  Afghanistan  identify  the  districts  in  which  it  intends  to  implement  post¬ 
transition  development  projects,  develop  transition  plans  for  districts  in  which  it  plans  to  implement 
future  projects,  and  close-out  programs  for  districts  where  it  will  not  implement  new  projects. 

On  October  21,  201 1,  OTI  representatives  in  Afghanistan  developed  a  draft  set  of  disengagement  criteria 
in  response  to  recommendations  from  the  two  program  performance  reviews  conducted  by  OTI  and 
specifically  included  criteria  under  which  ASI-East  may  discontinue  stability  programming  in  a  district. 
However,  the  draft  lacks  target  indicators  or  benchmarks  that  could  provide  the  basis  for  applying  the 
broad  criteria  described  in  the  draft  criteria.  These  target  indicators  or  benchmarks  should  be  tied  to  the 
impact  and  overall  stability  measures  which  DAI  and  Altai  Consulting  are  currently  developing. 

The  OTI  official  who  drafted  the  disengagement  criteria  was  subsequently  re-assigned;  however,  OTI’s 
current  Country  Representative  noted  the  document  has  not  yet  been  vetted  or  discussed  with  Mission  or 
military  colleagues  at  post.  OTI  officials  noted  that  these  draft  exit  criteria  will  be  finalized  in  connection 
with  the  new  ASI  task  order. 

OTI  officials  noted  that  an  18-month  district  exit  strategy  has  not  been  developed  and  that  no  other 
transition  plans  have  been  put  in  place  by  the  embassy.  OTI  officials  noted  that  an  exit  strategy  will  be 
developed  under  the  follow-on  task  order  to  ASI-East  that  will  combine  OTI  stability  programming 
efforts  for  both  the  South  and  East  under  a  single  task  order,  which  was  awarded  in  March  2012,  for  a 
3  year  period.  The  development  of  this  exit  strategy  will  need  to  be  consistent  with  any  final 
disengagement  criteria  OTI  develops  and  carefully  coordinated  with  ongoing  civilian  and  military 
transition  planning  efforts. 

The  issue  of  disengagement  criteria  and  exit  strategy  are  linked  to  current  and  future  implementation  of 
the  DSF  methodology.  Responsibility  for  further  development  and  evaluation  of  the  DSF  has  been 
transferred  from  OTI  to  USAID  Afghanistan’s  Stabilization  Unit,  which  will  now  provide  DSF  training 
and  future  assessments  of  and  enhancements  to  the  methodology.  A  Stabilization  Unit  official  noted  that 
the  Stabilization  Unit  intends  to  include  a  review  of  the  DSF  under  USAID’s  Measuring  Impact  of 
Stabilization  Initiatives  (MISTI)  program.  MISTI  is  a  $  15-million,  three-year  program  that  began  in 
March  2012,  which  will  provide  third-party  impact  assessment  and  evaluation  services  for  the 
Stabilization  Unit.  As  part  of  this  effort,  the  contractor  will  be  expected  to  evaluate  the  implementation  of 
the  DSF  and  draw  lessons  learned. 


17 

USAID  Office  of  Inspector  General,  Audit  of  USAID /Afghanistan ’s  Afghanistan  Stabilization  Initiative  for  the 
Southern  Region ,  Audit  Report  No.  F-306-12-001-P,  November  13,  201 1. 
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CONCLUSIONS 


As  a  key  building  block  of  the  U.S.  COIN  strategy,  stabilization  programs  such  as  ASI  must  succeed  if 
the  United  States  is  to  achieve  its  short  and  long-term  military  and  economic  support  drawdown  goals. 

As  the  pilot  implementation  of  the  DSF  methodology,  the  ASI-East  program  provides  a  critical 
opportunity  to  extract  lessons  learned  for  application  to  follow-on  stabilization  awards  which  will  use  the 
DSF  methodology.  Key  challenges  facing  the  implementation  of  ASI-East’ s  stabilization  efforts  and 
related  follow-on  efforts  include  (1)  relatively  low  project  spending  levels  and  high  operating  costs, 

(2)  certain  procedural  deficiencies  with  key  DAI  business  systems  and  certain  questioned  costs,  (3)  the 
implementation  and  assessment  of  the  DSF  programming  methodology,  and  (4)  the  lack  of  approved 
district-level  disengagement  criteria  and  a  country- wide  exit  strategy  for  OTI’s  programming  efforts. 
Additionally,  it  is  noted  that  the  ASI-East  program  has  not  yet  transitioned  any  target  districts  from  the 
“hold”  phase  to  the  “build”  phase  of  the  COIN  strategy. 


RECOMMENDATIONS 

To  help  ensure  that  the  ASI-East  program  is  implemented  within  the  terms  of  OTPs  IQC  and  the  ASI- 
East  task  order  and  applicable  cost  principles,  SIGAR  recommends  that  USAID/DCHA/OTI  in 
Washington,  D.C.,  in  conjunction  with  OTI  Kabul  mission  staff: 

1 .  Review  the  ASI-East  program  spending  patterns  to  determine  if  the  current  spending  levels 
represent  a  reasonable  balance  between  program  spending  versus  operating  costs. 

2.  Instruct  the  contractor  to  address  and  correct  the  identified  timekeeping  and  billing  internal 
control  weakness  and  procedural  shortcomings. 

To  ensure  that  ASI-East  program  costs  are  allowable,  allocable,  and  reasonable,  we  recommend  that  the 
Contracting  Officer  in  Washington,  D.C.: 

3.  Review  the  more  than  $590,000  in  questioned  costs  outlined  below  and  recover  these  funds  as 
deemed  appropriate: 

a.  The  $500,000  in  potentially  unallowable  costs  associated  with  DATs  sub-contract  award  to 
Altai  Consulting  for  monitoring  and  evaluation  services. 

b.  The  $3,400  in  unallowable  shipping  costs  associated  with  a  different  DAI  contract. 

c.  The  $50,000  in  wasteful  spending  on  insurance  cost  for  non-operational  armored  vehicles. 

d.  The  $38,500  in  wasteful  spending  on  insurance  for  leased  armored  vehicles. 

To  improve  ASI-East  program  performance  oversight,  monitoring,  and  evaluation  systems  and  assess  the 
overall  utility  and  success  of  the  DSF  programming  model,  SIGAR  recommends  that  USAID/DCHA/OTI 
in  Washington,  D.C.,  in  conjunction  with  OTI  Kabul  mission  staff  and  the  Stabilization  Unit  Chief,  as 
appropriate: 

4.  Incorporate  an  assessment  of  activity  outcome  results  into  level  1/output  measurement  of  the  DSF 
monitoring  and  evaluation  methodology  and  perform  level  3/overall  stability  monitoring  and 
evaluation  when  entering  new  districts. 

5.  Provide  input  to  the  Stabilization  Unit  Chief  to  produce  an  interim  and  final  lessons  learned 
summary  of  the  ASI-East  program’s  implementation  of  the  DSF  to  be  considered  by  senior 
USAID,  military,  and  State  Department  officials  as  future  decisions  are  made  regarding  the 
application  of  the  DSF  methodology  to  other  stabilization  programs. 
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To  help  ensure  future  progress  toward  transitioning  districts  to  the  “build”  phase  of  the  COIN  strategy, 
SIGAR  recommends  that  USAID’s  Bureau  for  Democracy,  Conflict,  and  Humanitarian  Assistance/Office 
of  Transition  Initiatives  in  Washington,  D.C.,  in  conjunction  with  OTI  Kabul  mission  staff: 

6.  Finalize  draft  disengagement  criteria  to  facilitate  programming  decisions  at  the  district  level  and 
develop  an  exit  strategy  to  guide  overall  OTI  programming  decisions  at  the  country  level. 


COMMENTS 

In  commenting  on  a  draft  of  this  report,  USAID/OTI  generally  concurred  with  the  recommendations  and 
noted  the  steps  that  it  has  taken  or  will  take  to  address  them.  USAID  also  provided  some  general  and 
technical  comments  and  clarifications,  particularly  regarding  the  role  of  ASI-East  in  the  larger  COIN 
strategy,  which  we  incorporated  into  the  final  report,  as  appropriate.  USAID’s  comments  and  our 
response  to  these  comments  are  reproduced  in  appendix  III. 

In  response  to  recommendation  one,  USAID/OTI  concurred  with  the  recommendation  and  stated  that 
reviews  have  taken  place  during  the  course  of  the  program  and  will  continue  for  the  remainder  of  the 
program.  OTI  provided  clarification  of  the  terminology  we  used  in  the  report  and  recommendation 
regarding  our  characterization  of  programming  spending  levels.  In  response,  we  incorporated  the 
distinction  between  program  costs  versus  operational  costs,  as  opposed  to  program  costs  versus  overhead 
costs. 

Regarding  the  second  recommendation,  USAID/OAA  requested  clarification  of  the  guidance  that  was 
used.  We  provided  the  guidance  that  we  used  in  SIGAR  comment  six  in  appendix  III. 

In  response  to  recommendation  three,  USAID/OAA  concurred  with  the  recommendation  and  has  taken  or 
will  take  steps  to  recover  any  money  that  is  determined  unallowable.  USAID  specifically  stated  that 
(1)  the  $500,000  in  questioned  contract  costs  was  allowable  and  no  recovery  action  was  required;  (2)  the 
$3,400  in  questioned  shipping  costs  has  already  been  recovered;  (3)  the  $50,000  in  questioned  insurance 
costs  is  currently  being  reviewed  by  the  contracting  officer;  and  (4)  the  $38,500  in  questioned  insurance 
costs  is  currently  being  reviewed  by  DAI  and  any  overpayment  adjustments  made  by  the  insurance  carrier 
will  be  recovered. 

In  response  to  recommendation  four,  USAID/OTI  concurred  that  level  3  (overall  stability)  analysis  should 
be  performed  when  entering  a  new  district,  but  did  not  concur  that  outcomes  should  be  added  to  level  1 
analysis  because  they  do  not  have  ownership  of  the  DSF.  In  addition,  USAID/OTI  stated  that  the  ASI 
program  has  more  robust  analysis  at  the  level  1  (output)  stage,  which  provides  some  level  of  outcome 
monitoring,  and  is  looking  at  options  to  most  effectively  measure  outputs,  outcomes,  impact,  and  overall 
stability  in  future  programming  and  further  noted  that  MISTI  will  facilitate  this  process.  We  agree  that 
OTI  no  longer  has  ownership  of  the  DSF  and  updated  our  recommendation  to  also  include  the 
Stabilization  Unit,  given  the  collaborative  relationship  between  the  DSF  coordinator  and  the  Stabilization 
Unit. 

In  response  to  recommendation  five,  USAID/OTI  concurred  that  lessons  learned  from  the  DSF  should  be 
shared  with  other  stakeholders  and  implementers.  USAID/OTI  did  not  concur  with  the  recommendation 
to  develop  a  lessons  learned  DSF  summary,  stating  it  was  a  duplicative  effort  under  the  Stabilization 
Unit’s  ongoing  efforts.  We  believe  that  OTI  remains  the  appropriate  office  to  produce  a  comprehensive 
DSF  lessons  learned  study,  given  its  role  and  mission.  However,  we  updated  our  recommendation  to 
include  the  Stabilization  Unit,  given  its  ongoing  effort  with  the  MISTI  program  to  compile  lessons 
learned  on  all  programs. 

In  response  to  recommendation  six,  USAID/OTI  concurred  that  a  disengagement  strategy  at  the  district 
level  and  an  exit  strategy  at  the  country  level  should  be  developed.  USAID/OTI  noted  that  it  has  begun  to 
draft  district  disengagement  criteria  and  will  finalize  the  criteria  under  a  new  OTI  program.  OTI  stated 
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that  it  would  ensure  that  the  disengagement  criteria  are  finalized  prior  to  May  2013,  allowing  for  an 
informed  assessment  of  when  OTI  should  disengage  from  a  particular  district/area  of  operation. 

Decisions  to  disengage  will  take  into  account  whether  OTI  can  guide  or  support  follow  on  programming 
and  the  Administrator’s  Stabilization  Guidance  and  will  include  discussions  pertaining  to  GIRoA 
transition  plans  for  provinces.  Additionally,  USAID  will  work  with  the  Embassy  mission  to  determine  an 
appropriate  country  level  disengagement  strategy. 
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APPENDIX  I:  SCOPE  AND  METHODOLOGY 


This  report  provides  the  results  of  the  Office  of  the  Special  Inspector  General  for  Afghanistan 
Reconstruction’s  audit  of  USAID’s  Afghanistan  Stabilization  Initiative -East  program  to  implement  small- 
scale  community  development  and  media  activities  to  local  implementers  in  target  districts  for  the 
36  month  period  from  June  26,  2009  through  June  25,  2012.  This  report  assesses  to  what  extent 

(1)  expended  ASI-East  costs  were  allowable,  allocable,  and  reasonable;  (2)  performance  oversight, 
monitoring,  and  evaluation  systems  have  been  implemented;  and  (3)  progress  has  been  made  toward 
transitioning  districts  to  the  “build”  phase  of  the  COIN  strategy.  We  reviewed  documents  covering  the 
period  June  2009  through  June  2012. 

To  determine  if  incurred  project  costs  were  allowable,  allocable,  and  reasonable,  we  examined  invoices, 
evaluated  timekeeping  and  billing  policies  and  procedures  related  to  internal  controls,  and  had  our 
forensic  team  analyze  DATs  financial  data.  With  the  invoice  review,  we  judgmentally  selected 
377  invoices  from  6  other  direct  cost  expense  types  for  testing.  For  each  invoice,  we  determined  whether 
adequate  support  documentation  existed  and  required  approvals  were  obtained.  In  addition,  to  access  the 
reliability  of  computer-processed  data,  we  (1)  interviewed  officials  to  discuss  the  reliability  of  the  data; 

(2)  conducted  testing  and  checked  for  missing  data,  erroneous  or  incomplete  entries,  and  duplicates;  and 

(3)  compared  system  generated  data  with  documents  in  the  project  files.  For  the  timekeeping  and  billing 
systems,  we  interviewed  DAI  staff  in  Kabul,  Afghanistan  to  gain  an  understanding  of  how  the  policies 
and  procedures  worked,  obtained  policy  and  procedure  manuals,  and  performed  walk  through  testing.  We 
compared  the  results  and  noted  any  discrepancies.  We  also  reviewed  the  reasonableness  of  DATs 
application  of  indirect  cost  rates  and  fixed  fees.  Additionally,  we  obtained  and  reviewed  DATs  external 
auditors’  management  report  for  the  two  most  recent  audits  performed  and  noted  any  internal  control 
weaknesses  identified.  We  investigated  any  anomalies  to  determine  if  the  costs  we  properly  recorded. 

To  determine  whether  performance  oversight,  monitoring  and  evaluation  systems  had  been  implemented, 
we  examined  the  task  order  to  determine  the  reporting  requirements.  We  interviewed  USAID  officials  in 
Kabul,  Afghanistan  and  the  contracting  officer  in  Washington,  DC  to  determine  the  reports  that  were 
being  sent  and  received  and  compared  them  to  the  reporting  requirements.  Additionally,  we  interviewed 
the  contracting  officer  and  COTR  in  Washington,  DC  to  determine  how  they  provide  oversight  to  the 
program.  We  examined  20  judgmentally  selected  activity  folders  to  determine  if  they  had  all  the 
necessary  reporting  documentation,  the  proper  USAID  signatures  noting  the  activity  was  complete  and 
documentation  of  completion.  We  also  examined  DATs  performance  management  system  to  determine  if 
it  met  the  task  order  requirements  and  if  it  was  performing  adequately.  We  examined  USAID’s  third 
party  evaluator  and  the  reports  produced  to  determine  what  oversight  and  performance  reporting  was 
being  provided.  Based  on  the  third  party  evaluator’s  reports,  we  determined  what  program  challenges 
were  being  encountered  and  how  USAID  and  DAI  were  addressing  the  challenges. 

To  determine  if  progress  has  been  made  toward  transitioning  districts  to  the  “build”  phase  of  the  COIN 
strategy,  we  examined  the  task  order  to  identify  any  transition  requirements  and  compared  it  to  work 
performed  by  DAI.  We  also  interviewed  USAID  and  DAI  officials  to  note  if  any  districts  covered  by  the 
program  had  transitioned  into  the  next  phase.  Finally,  we  analyzed  the  DAI  90  activities  completed  as  of 
September  2011. 

We  reviewed  the  financial  data,  including  invoice  and  payment  data,  provided  by  USAID  and  DAI  and 
determined  that  the  data  provided  was  sufficiently  reliable  for  the  purposes  of  our  audit.  We  also  assessed 
the  adequacy  of  internal  controls  over  DATs  timekeeping  and  billing  systems  and  procedures  through 
tests  of  selected  transactions,  interviews  and  physical  observations. 

We  conducted  work  in  Kabul,  Afghanistan,  and  Washington,  D.C.,  from  May  2011  to  June  2012,  in 
accordance  with  generally  accepted  government  auditing  standards.  These  standards  require  that  we  plan 
and  perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  the  evidence  obtained  provides  a 
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reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives.  The  audit  was  conducted 
by  the  Office  of  the  Special  Inspector  General  for  Afghanistan  Reconstruction  under  the  authority  of 
Public  Law  No.  1 10-181,  as  amended,  the  Inspector  General  Act  of  1978,  and  the  Inspector  General 
Reform  Act  of  2008. 


SIGAR  Audit-12-1 1  Contractor  Performance  and  Oversight/Stabilization  Initiative 


Page  17 


APPENDIX  II:  PRELIMINARY  MONITORING  AND  EVALUATION  RESULTS 


Tables  I  and  II  outline  DAI  and  Altai  Consulting  monitoring  and  evaluation  systems  and  preliminary 
results  through  January  2012. 


Table  I:  DAI  Monitoring  and  Evaluation  Preliminary  Results 


Level 

Description 

Preliminary  Results 

Level  1 

(program  outputs) 

DAI  collects  output  information  for  all 
awarded  activities  in  its  its  TAMIS 
database  and  generates  Final 

Monitoring  Reports  for  each  completed 
activity.  A  DAI  monitoring  and 
evaluation  staff  member  prepares  a 
lessons  learned  report  using  data  from 
the  TAMIS  system  for  a  sample  of  90 
completed  activities. 

After  each  activity  is  completed,  DAI 
produces  a  Final  Monitoring  Report,  which 
is  used  to  collect  data  at  the  close  of  each 
activity.  This  report  provides  detailed 
information  about  the  outputs  and  activities, 
the  implementation  process,  challenges 
encountered  and  lessons  learned,  the 
grantee’s  performance,  and  project 
stakeholders’  assessments  of  the  activity’s 
overall  performance. 

In  September  201 1 ,  DAI  completed  its  first 
lessons  learned  report  that  analyzed  90 
closed  activities.  This  report  found  that, 
overall,  activities  are  successful  at 
achieving  intended  purposes.  The  report 
also  highlighted  methods  of  implementation 
that  were  successful  and  should  be  used  in 
the  future  to  ensure  that  other  activities 
achieve  their  goals  and  are  successful. 

Level  2 

(program  impact) 

DAI  started  an  assessment  of  the 
impact  of  clusters  of  activities  against 
specific  sources  of  instability  through 
case  studies.  These  case  studies 
examine  the  outcomes  of  projects  on 
district  areas.  The  impact  assessment 
is  entirely  qualitative  and  is  gathering 
data  from  a  range  of  stakeholders 
including  program  staff,  district  teams, 
residents  of  the  district,  and 
stakeholders  in  stability  working  groups 
and  provincial  reconstruction  teams. 

A  DAI  monitoring  and  evaluation  staff 
member  managed  the  research  and 
production  of  the  case  studies  which 
are  designed  to  note  observable 
changes  in  stability  and  identify  which 
program  strategies  have  and  have  not 
succeeded. 

DAI  released  the  preliminary  results  of  its 
impact  assessment  in  four  districts  in 

January  2012.  In  5  of  9  cases,  ASI-East 
programming  was  found  to  be  a  positive 
factor.  In  one  case,  ASI-East  programming 
was  found  to  be  a  mitigating  factor.  In  3 
cases,  the  data  could  not  establish  a 
positive  or  negative  effect. 

Additionally,  the  assessment  found  that  the 
DSF  methodology  allowed  for  identifying 
sources  of  instability  and  adjusting 
programming  to  meet  address  them.  The 
assessment  recommended  (1 )  that  ASI- 
East  programming  should  consider  the 

COIN  status  of  districts  and  restrict  high 
visibility  projects  until  districts  are  moved 
from  the”shape”  phase,  (2)  more 
coordination  with  the  military  so  that 
projects  follow  military  operations  for  better 
effectiveness,  and  (3)  increased  community 
involvement  of  government  officials. 

Level  3 

(overall  progress  toward 
stability) 

DAI,  in  concert  with  OTI,  developed 
seven  district-level  stability  indicators 
within  the  DSF  and  selected  methods 
for  their  measurement.  The  purpose  is 
not  to  measure  direct  impact  of  DAI 
projects,  but  rather  to  identify  overall 
stability  in  the  areas  in  which  DAI 
operates.  To  standardize  data 
collection,  DAI  awarded  a  subcontract 
to  an  Afghan  research  firm,  RSI 
Consulting,  to  collect  data  on  the  7 
indicators  across  10  districts. 

In  October  2011,  RSI  Consulting  provided 

DAI  with  a  final  report  for  ten  districts 
indicating  that  overall  stability  in  all  ten 
districts  is  negative  based  on  the  key 
indicators  of  stability.Specifically,  the 
decline  in  stability  is  attributed  to  security 
issues  within  districts,  a  lack  of  confidence 
in  the  local  and  national  government  and 
underlying  cultural  differences  between 
locals  and  outsiders. 

Source:  SIGAR  Analysis  of  DAI  M&E  Documents 
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Table  II:  Altai  Consulting  Monitoring  and  Evaluation  Preliminary  Results 
Category  Description  Preliminary  Results 


Cycle  1 
(monitoring) 


OTI  contracted  with  QED,  which 
subcontracted  with  Altai  Consulting,  to 
perform  third  party  monitoring  of  DAI’s 
projects  in  a  new  innovative  way. 

In  October  2010,  Altai  issued  an 
inception  report  which  summarized  a 
plan  to  develop  methodological 
guidelines  for  M&E.  Altai  primarily 
relied  upon  physical  project  visits, 
perception  surveys  and  case  study 
reviews  of  completed  projects. 


In  March  2011,  Altai  released  its  first  cycle 
report  that  examined  two  districts  in  the 
East.  This  report  found  that  across  all 
districts,  ASI  projects  generally  addressed 
the  grievances  of  the  community,  mostly 
addressed  the  priority  problems  of  the 
community,  built  governance  capacity,  and 
were  being  implemented  according  to  their 
designed  plan. 


Cycle  2 

(monitoring/impact/overall 

stability) 


In  cycle  2,  Altai  had  started  an 
assessment  of  the  impact  of  clusters  of 
activities  against  specific  sources  of 
instability  through  case  studies.  These 
case  studies  examine  the  outcomes  of 
projects  on  district  areas.  The  impact 
assessment  is  entirely  qualitative  and  is 
gathering  data  from  a  range  of 
stakeholders  including  program  staff, 
district  teams,  residents  of  the  district, 
and  stakeholders  in  stability  working 
groups  and  provincial  reconstruction 
teams. 


In  July  201 1 ,  Altai  released  its  cycle  2 
report  for  the  same  two  districts  examined 
previously.  The  report  found  that  overall 
projects  were  generally  received  well  from 
the  communities.  The  report  provided 
insights  into  community  perceptions, 
provided  lessons  learned,  and  noted  that 
projects  need  a  better  association  with  the 
Afghan  government  for  increased 
credibility. 


Cycles  3  and  4  Cycles  3  and  4  will  apply  the  same  Reports  planned  in  two  districts  (plus  one 

(monitoring/impact/overall  methodology  used  in  cycle  2,  with  new  district  and  five  new  villages)  in 

stability)  further  development  of  tools.  January/February  2012  and  May/June 

2012. 


Source:  SIGAR  Analysis  of  Altai  Consulting  monitoring  and  evaluation  documents. 
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APPENDIX  III:  COMMENTS  FROM  THE  U.S.  AGENCY  FOR  INTERNATIONAL 
DEVELOPMENT 


MEMORANDUM 


June  6,  2012 


TO: 


Jenniffer  F.  Wilson.  Deputy  Assistant  IG  for  Audits 
Special  Inspector  General  for  Afghanistan  Reconstruction 
(SIGAR) 


FROM:  Robert  Jenkins,  Director,  DCHA/OTI 


SUBJECT:  Draft  SIGAR  Report  entitled  “Progress  Made  Toward 


Increased  Security  under  USAID’s  Afghanistan  Stabilization 
Initiative-East  Program,  but  Transition  to  Long  Term 
Development  Efforts  Not  Yet  Achieved”,  (SIGAR  12-11) 


SIGAR  Transmittal  email  dated  5/16/2012 


REF: 


Thank  you  for  providing  the  Bureau  for  Democracy,  Conflict  and 
Humanitarian  Assistance’s  Office  of  Transition  Initiatives  (DCHA/OTI) 
with  the  opportunity  to  review  the  subject  draft  audit  report.  Discussed 
below  are  our  comments  on  the  recommendations  and  contents  of  the 
report. 

USAID’S  (DCHA/OTI)  GENERAL  COMMENTS  ON  THE  DRAFT 
REPORT 

SIGAR’s  focus  on  progress  toward  increased  security  in  the  title  of  the 
audit  does  not  accurately  reflect  the  objectives  of  Afghanistan 
Stabilization  Initiative  (ASI)  East.  Civilian  security  is  only  one  of  the 
indicators  under  the  District  Stability  Framework  (DSF)  that  contributes 
to  an  overall  outcome  of  stability.  Under  the  DSF,  civilian  security  and 
perceptions  of  security,  along  with  five  other  indicators,  are  used  to 
measure  stability  in  a  district.  OTI  programming  does  not  directly 
address  security;  rather  OTI  coordinates  closely  with  Coalition  and 
Afghan  security  forces  to  ensure  that  security  conditions  are  appropriate 
for  stabilization  programming  in  an  area. 

While  ASI  does  serve  as  a  tool  for  the  U.S.  Government  (USG)  to  “hold” 
areas,  the  task  order  outlines  that  longer  term  development  is  not  within 
the  scope  of  the  ASI  task  order. 


Tel:  202-216-6288  /  0700-108-001 

U.S.  Agency  for  International  Development  Email:  kabulusaidinfomnation@usaid.qov 

Great  Massoud  Road  http://afQhanistan.usaid.qov 

Kabul,  Afghanistan 


See  SIGj 

commei 

*\R 

nt  1. 

See  SIGAR 
comment  2. 
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OTI  does  not  dispute  the  accuracy  of  SIGAR’s  finding  that  despite  nearly 
three  years  of  program  efforts,  none  of  ASI-Easf  s  target  districts  have 
transitioned  to  the  “build”  phase  of  the  U.S.  Counterinsurgency 
(COIN)  strategy.  However,  districts  in  Wardak  and  Kunar  where  ASI- 
East  has  programmed  were  announced  on  May  13,  2012  by  President 
Karzai  as  Tranche  3  designated  transition  areas,  meaning  these  areas  were 
identified  as  ready  for  transfer  to  GIRoA  responsibility.  In  Wardak,  OTI 
will  be  transitioning  stabilization  programming  to  SIKA,  a  program 
intended  to  bridge  stabilization  to  a  long  term  development  program. 

OTI  would  like  to  clarify  the  role  ASI-East  plays  in  the  larger  COIN 
strategy.  The  ASI-East  aims  to  support  the  transition  from  “hold”  to 
“build”  in  the  districts,  where  its  programming  is  a  small  component  of 
the  COIN  strategy,  and  as  such,  the  program  by  itself  is  not  in  a  position 
to  independently  drive  the  successful  transition  from  hold  to  build  for 
entire  districts  or  make  decisions  regarding  the  placement  of  long  term 
development  programming. 

As  recognized  by  the  Administrator’s  Stabilization  Guidance  issued  in 
January  201 1,  a  locality  that  is  stable  today  may  be  unstable  tomorrow, 
and  USAID  needs  to  clearly  identify  how  a  program  supports  the 
advancement  of  a  community  along  the  stability  continuum.  Based  on 
this  guidance,  ASI  has  used  the  DSF  to  identify  activities  to  program 
against  sources  of  instability  in  an  effort  to  solidify  stabilization  gains  for 
ASI-East  districts. 

OTI  is  committed  to  sharing  lessons  learned  about  stabilization  program 
implementation  using  the  DSF  and  agrees  with  SIGAR’s  assessment  of 
the  importance  of  continuing  to  share  these  lessons  with  the  wider  USG 
audience.  While  OTI  will  continue  to  engage  in  these  discussions  and 
provide  additional  information  and  reports  as  requested  by  the 
Stabilization  Unit,  the  DSF  is  not  owned  by  OTI,  and  as  such,  OTI  cannot 
take  unilateral  actions  to  amend  the  framework  without  the  involvement 
of,  among  others,  the  Stabilization  Unit  at  USAID/Afghanistan  that  will 
lead  support  for  DSF  going  forward.  DSF  is  a  tool  used  by  several 
USAID  programs,  as  well  as  used  for  USG  civilian  and  military  planning 
and  stabilization  efforts  in  Afghanistan  and  other  countries. 

While  the  recommendations  do  not  directly  suggest  any  action  regarding 
the  administrative  and  management  issues  relating  to  program  oversight, 
OTI  feels  it  is  important  to  address  the  three  issues  outlined  in  Table  1 
shown  in  Page  8  of  the  report.  With  regard  to  two  of  the  issues,  Data 
Validity  Concerns  and  Fraud  Reporting,  corrective  actions  have  already 
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been  taken  as  discussed  in  the  following  paragraphs.  On  the  third  issue, 
Subcontract  Oversight,  OTI  has  communicated  the  concerns  to  the  ASI- 
East  implementing  partner,  Development  Alternatives  Inc.  (DAI),  and 
will  continue  to  work  with  DAI  to  ensure  the  costs  incurred  under  the 
Edinburgh  International  (El)  sub-contract  are  allowable,  allocable  and 
reasonable. 

OTI  concurs  with  SIGAR’s  assessment  that  data  validation  should  occur 
at  the  field  level,  and  that  more  frequent  and  in-depth  data  verification 
should  take  place  to  ensure  that  DAI  maintains  and  uses  the  USAID/OTI 
activity  database  as  the  “backbone”  to  generate,  develop,  manage, 
monitor,  evaluate  and  report  on  program  activities.  The  Contracting 
Officer’s  Representative  (COR),  based  in  Washington  DC,  will  continue 
to  perform  weekly  data  validity  checks.  The  OTI  field  team  has  been 
performing  additional  routine  checks  as  a  result  of  SIGAR’s  findings. 

As  outlined  in  the  report,  DAI  has  already  changed  its  policy  on  reporting 
fraud  based  on  SIGAR’s  recommendations.  DAI  will  notify  the 
Contracting  Officer  (CO)  of  any  future  incident  regardless  of  the  amount. 
Within  its  Afghanistan  Operations  Office,  which  supports  the  ASI-East 
program,  DAI  has  maintained  a  full-time  auditor,  a  fraud  prevention 
team,  and  compliance  managers.  As  outlined  in  the  report,  two  of  the 
incidents  of  fraud  were  never  billed  to  USAID,  and  in  the  third  case,  the 
charge  was  reimbursed  to  OTI,  and  therefore  does  not  require  further 
action. 

PART  I:  COMMENTS  ON  SIGAR  RECOMMENDATIONS 

To  help  ensure  that  the  ASI-East  program  is  implemented  within  the 
terms  of  OTPs  IQC  and  the  ASI-East  task  order  and  applicable  cost 
principles,  SIGAR  recommends  that  USAID ’s  Bureau  for  Democracy, 
Conflict,  and  Humanitarian  Assistance/Office  of  Transition  Initiatives 
in  Washington,  D.C.,  in  conjunction  with  OTI  Kabul  mission  staff: 

1.  Review  the  ASI-East  program  spending  patterns  to  determine  if  the 
current  spending  levels  represent  a  reasonable  balance  between 
program  spending  versus  overhead  costs. 

DCHA/OTI  Comments: 

OTI  concurs  with  this  recommendation.  This  review  has  taken  place 
during  the  course  of  the  program  and  OTI  will  ensure  to  continue  to 
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review  program  and  operations  spending  for  the  remainder  of  the  task 
order. 

OTI  would  like  to  clarify  the  terminology  used  in  Recommendation  1. 
The  spending  levels  reviewed  by  SIGAR,  and  outlined  in  the  report, 
represent  program  versus  operational  costs,  as  opposed  to  program 
versus  overhead  costs.  The  distinction  between  these  two  is  of 
significant  importance,  as  overhead  is  typically  referenced  narrowly  to 
mean  the  costs  of  the  headquarters  of  an  implementing  partner. 

In  the  case  of  ASI-East,  the  operational  costs  include  all  spending  that 
is  required  to  implement  and  manage  a  program  in  Afghanistan  (local 
Afghan  and  expatriate  staff  implementing  and  monitoring  stabilization 
activities  in  ASI-East’ s  target  districts),  in  addition  to  overhead. 
USAID,  in  an  action  memo  dated  August  1,  2006  for  the  Counselor, 
has  determined  that  operational  or  administrative  costs  (including 
normal  costs  of  doing  business  and  overhead  type  costs)  incurred  in 
support  of  a  contractor  implementing  USAID-funded  projects  are 
considered  to  be  program  costs.  Therefore,  computing  an  “implied 
spending  ratio”  of  program  activities  to  operational  costs  is  not  valid. 

OTI  has  reviewed  the  program  spending  patterns  through  financial 
forecasting  processes  conducted  in  coordination  with  DAI,  and  more 
directly  through  the  bi-monthly  review  of  invoices,  over  the  course  of 
the  program.  DAI  and  OTI  have  discussed  program  and  operational 
spending  at  meetings  held  both  in  Kabul  and  Washington  DC,  most 
recently  in  advance  of  the  incremental  funding  completed  at  the 
beginning  of  2012.  An  additional  review  is  currently  taking  place 
with  an  updated  pipeline  provided  by  DAI  to  OTI  at  the  beginning  of 
May  2012. 

2.  Instruct  the  contractor  to  address  and  correct  the  identified 

timekeeping  and  billing  internal  control  weakness  and  procedural 
shortcomings. 

DCHA/OTI  Comments: 
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OTI  has  reviewed  the  timekeeping  and  billing  internal  controls 
deficiencies  outlined  by  SIGAR  and  requests  clarification  on  which 
FAR,  AIDAR,  ADS  guidance  was  used  to  determine  the  deficiencies 
in  DAPs  processes  to  ensure  OTI  and  DAI  can  more  adequately 
address  the  noted  deficiencies.  Once  this  has  been  established,  the 
cognizant  CO  will  require  DAI  to  formally  advise  DCHA/OTI  of 
actions  it  has  taken  or  plans  to  take  to  address  the  internal  control 
weaknesses.  Upon  receipt  of  information,  the  CO  will  make  a 
determination  on  the  adequacy/acceptability  of  DAI’s  corrective 
actions  where  required. 

To  ensure  that  ASI-East program  costs  are  allowable ,  allocable,  and 
reasonable,  SIGAR  recommends  that  the  Contracting  Officer  in 
Washington,  D.  C. : 

3.  Review  the  $592,000  in  questioned  costs  outlined  below  and  recover 
these  funds  as  deemed  appropriate: 

a .  The  $500,000  in  potentially  unallowable  costs  associated  with  DAIfs 
sub-contract  award  to  Altai  Consulting  for  monitoring  and  evaluation 
services . 

b.  The  $3,400  in  unallowable  shipping  costs  associated  with  a  different 
DAI  contract 

c.  The  $50,000  in  wasteful  spending  on  insurance  cost  for  non- 
operational  armored  vehicles. 

d.  The  $38,500  in  wasteful  spending  on  insurance  for  leased  armored 
vehicles. 

M/OAA  Comments: 

The  Office  of  Acquisition  and  Assistance  in  the  Bureau  for 
Management  (M/OAA)  at  USAID/Washington  agrees  with  the 
recommendation  and  has  taken  the  following  actions: 

Item  (a):  The  DCHA/OTI  Contracting  Officer  has  reviewed  the 
allowability  of  the  $500,000  questioned  costs  associated  with  DAI’s 
sub-contract  award  to  Altai  for  monitoring  and  evaluation  services, 
and  has  determined  that  these  costs  are  allowable,  per  FAR  Clause  52- 
216-7,  Allowable  Cost  and  Payment  under  the  base  SWIFT  III  IQC. 
Therefore,  no  recovery  action  is  required. 
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Item  (b):  The  $3,400  questioned  shipping  costs  associated  with  a 
different  DAI  contract  were  credited  to  OTI  in  DATs  February  2012 
invoice  and  as  such,  have  already  been  recovered. 

Item  (c):  The  Contracting  Officer  is  conducting  a  review  to  determine 
the  allowability  of  the  $50,000  insurance  cost  and  if  any  recovery  of 
funds  is  required. 

Item  (d):  The  Contracting  Officer  has  reviewed  the  $38,500 
questioned  insurance  costs  for  leased  armored  vehicle.  DATs 
insurance  carrier  bills  DAI  in  December  of  each  year  for  the  expected 
insurance  costs  of  the  upcoming  year.  The  cost  is  based  on  the 
number  of  vehicles  currently  in  the  project’s  possession.  As  vehicles 
are  added,  dropped,  or  replaced  during  the  coverage  year,  DAI  reports 
the  adjustments  to  the  insurance  carrier  accordingly.  In  April/May  of 
the  next  year,  the  insurance  carrier  reconciles  the  costs  for  the 
previous  year,  crediting  over-payment  or  billing  for  under-payment. 

For  ASI-East,  DAI  received  a  reconciled  invoice  in  May  2011  for  the 
insurance  costs  in  2010.  For  the  true  insurance  costs  for  2011,  DAI  is 
currently  working  with  the  insurance  company  to  reconcile  the 
insurance  charges.  Any  overpayment  will  be  credited  to  DAI  and  in 
turn,  to  USAID.  When  this  adjustment  occurs,  the  COR  for  ASI-East 
will  review  to  ensure  that  the  insurance  paid  were  for  vehicles  being 
utilized  by  the  project.  Final  determination  on  the  allowability  of  the 
cost  will  then  be  made  by  the  Contracting  Officer  and  appropriate 
recovery  action,  if  needed,  will  be  taken. 


4.  Incorporate  an  assessment  of  activity  outcome  results  in  level 
1/output  measurement  of  the  DSF  monitoring  and  evaluation 
methodology  and  perform  level  3/overall  stability  monitoring  and 
evaluation  when  entering  new  districts . 

DCHA/OTI  Comments: 

While  OTI  concurs  with  the  recommendation  to  perform  level 
3/overall  stability  monitoring  when  entering  new  districts,  OTI  does 
not  concur  with  the  recommendation  to  incorporate  activity  outcome 
results  in  level  1 /output  measurement. 
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Due  to  the  fact  that  the  DSF  methodology  is  not  owned  by  OTI  and  is 
used  by  several  different  entities,  OTI  cannot  take  unilateral  action  to 
amend  the  monitoring  and  evaluation  methodology  in  the  DSF  to 
include  activity  outcome  results  in  level  1 /output  measurement.  While 
OTI  cannot  make  such  a  change,  OTI  is,  however,  looking  at  options 
to  most  effectively  measure  outputs,  outcomes,  impact  and  overall 
stability  in  future  programming.  The  level  1  process  conducted  by 
ASI  is  more  robust  than  traditional  output  monitoring  that  only 
records  quantitative  outputs  of  activities. 

While  this  does  not  measure  short  term  change  after  each  activity,  it 
does  provide  a  comprehensive  understanding  of  the  success  of  the 
implementation  of  the  activity,  which  provides  some  parts  of  outcome 
monitoring.  In  addition,  the  level  2  process  (impact  assessment) 
conducted  by  ASI  tracks  the  impact  of  a  stability  objective  targeted  by 
a  number  of  small  grant  activities.  OTI  has  found  that  the  combination 
monitoring  and  evaluation  under  level  1  and  level  2  processes  is  a 
comprehensive  way  to  determine  the  effectiveness  of  OTI 
programming. 

OTI  agrees  that  assessing  overall  stability  when  entering  a  district  is  a 
best  practice  to  contribute  to  an  understanding  of  the  baseline  stability 
of  a  district,  and  to  facilitate  monitoring  changes  to  stability  in 
particular  districts  over  time.  The  new  Stabilization  Unit  Monitoring 
and  Evaluation  program,  Measuring  Impact  of  Stabilization  Initiatives 
(MISTI),  will  monitor  overall  stability  across  50-70  districts 
nationwide. 

The  data  will  be  used  to  develop  stability  baselines  and  identify  trends 
in  districts  where  there  is  USAID  Stabilization  programming, 
including  Community  Cohesion  Initiative  (CCI),  Stabilization  in  Key 
Areas  (SIKA)  and  Community  Development  Program  (CDP).  OTI’s 
new  program,  CCI,  was  only  recently  started  (March  2012),  making  it 
unrealistic  to  complete  a  comprehensive  study  of  overall  stability  upon 
entering  some  of  the  first  districts  in  which  it  will  operate.  As  MISTI 
becomes  operational  in  CCI  districts,  this  information  will  be  used  to 
inform  OTI  programmatic  decisions  and  program  effects. 
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5.  Produce  an  interim  and  final  lessons  learned  summary  of  the  ASF 
East  program ’s  implementation  of  the  DSF  to  be  considered  by  senior 
USAID ,  military ,  and  State  Department  officials  as  future  decisions  are 
made  regarding  the  application  of  the  DSF  methodology  to  other 
stabilization  programs . 

DCHA/OTI  Comments: 

OTI  agrees  with  the  recommendation  to  share  lessons  learned  from 
ASI-East’s  use  of  the  DSF  with  senior  USAID,  military  and  State 
Department  officials  to  inform  future  decisions.  However,  OTI  does 
not  concur  with  the  recommendation  to  “produce  an  interim  and  final 
lessons  learned  summary”  as  this  will  duplicate  USAID/Afghanistan 
Stabilization  Unit’s  ongoing  efforts  to  compile  DSF  lessons  learned. 

In  November  2011,  USAID/Afghanistan’s  Stabilization  Unit  put  in 
place  a  DSF  Coordinator  to  facilitate  DSF  training  and  lead  efforts  to 
expand  the  use  of  DSF  within  the  Mission.  The  DSF  Coordinator 
collaborates  with  the  Stabilization  Unit,  USAID  implementing 
partners  and  other  key  stakeholders  to  review  and  amend  the 
methodology  as  required.  The  DSF  Coordinator  is  also  responsible 
for  the  documentation  and  distribution  of  lessons  learned  to  key 
stakeholders  in  order  to  inform  the  future  application  of  the  DSF 
methodology. 

OTI  has  contributed,  both  formally  and  informally,  to 
USAID/Afghanistan’s  initiative  to  compile  lessons  learned  from 
program  design  and  implementation  using  DSF.  In  November  2011, 
ASI-East  hosted  a  “DSF  lessons  learned  summit”  where  key  lessons 
learned  from  DSF  implementation  and  training  were  shared  with 
USAID  and  military  colleagues. 

In  March  2012,  ASI-East  was  a  core  contributor  to  the  DSF  summit 
hosted  by  the  Stabilization  Unit.  USAID  and  its  implementing 
partners  were  in  attendance.  DAI  and  OTI  also  contributed  to  the 
report  drafted  by  Development  Transformations  at  the  request  of 
US  AID/ Washington’s  Civil  Military  Coordination  (formerly  Office  of 
Military  Affairs)  to  collect  lessons  learned  and  to  provide 
recommendations  on  the  standardization  of  DSF,  among  other 
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objectives.  These  three  specific  instances  are  in  addition  to  the 
informal  contributions  of  OTI  and  DAI  staff. 

The  plans  for  MISTI  include  conducting  an  evaluation  of  DSF  that 
will  be  shared  with  senior  USAID,  military  and  State  Department 
officials  to  inform  future  decisions  regarding  the  application  of  the 
DSF  methodology  to  other  stabilization  programs. 

Through  the  DSF  coordinator  at  the  Stabilization  Unit,  OTI  will 
continue  to  feed  into  DSF  methodology  discussions  for  the  MISTI 
evaluation  and  other  efforts  to  determine  DSF  best  practices.  Given 
that  USAID/Afghanistan  is  already  working  on  the  documentation  of 
DSF  lessons  learned,  we  believe  that  it  is  not  necessary  for  OTI  to 
produce  a  separate  summary. 

To  help  ensure  future  progress  toward  transitioning  districts  to  the 
“build” phase  of  the  COIN  strategy,  SIGAR  recommends  that  USAID 9 s 
Bureau  for  Democracy,  Conflict,  and  Humanitarian  Assistance/Office 
of  Transition  Initiatives  in  Washington ,  D.C.,  in  conjunction  with  OTI 
Kabul  mission  staff: 

6.  Finalize  draft  disengagement  criteria  to  facilitate  programming 
decisions  at  the  district  level  and  develop  an  exit  strategy  to  guide 
overall  OTI  programming  decisions  at  the  country  level 

DCHA/OTT  Comments: 

OTI  concurs  with  this  recommendation. 

As  noted  in  the  report,  OTI  has  already  undertaken  efforts  to  draft 
district  disengagement  criteria  and  will  finalize  the  criteria  under  the 
new  OTI  program,  CCI.  Where  appropriate,  OTI  will  aim  to  exit  a 
district/area  of  operation  between  12  and  18  months  from  the  start-up 
of  grant  activities  in  that  area.  With  new  programming  beginning 
under  the  new  task  order  in  May  2012,  OTI  will  ensure  that 
disengagement  criteria  is  finalized  prior  to  May  2013,  allowing  for  an 
informed  assessment  of  when  OTI  should  disengage  from  that 
particular  district/area  of  operation. 

The  criteria  and  decision  parameters  will  be  based  on  a  preponderance 
of  evidence,  not  complete  success,  and  will  take  into  account  whether 
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and  how  OTI  can  guide  and/or  support  any  follow  on  programming. 
Criteria  will  also  take  into  account  the  Administrator’s  Stabilization 
Guidance,  which  recognizes  that  since  the  operating  environment  is 
extremely  fluid,  USAID  must  poise  platforms  and  contingency 
resources  to  rapidly  expand  or  shift  programming  as  opportunities 
arise.  The  review  of  the  draft  exit  criteria  will  include  discussion  on 
how  to  set  conditions  for,  or  work  through,  GIRoA  transition  plans  for 
the  provinces. 

OTI  will  work  with  the  mission  to  determine  an  appropriate  country 
level  disengagement  strategy. 
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PART  II:  TECHNICAL  COMMENTS 

With  the  exceptions  noted  in  our  above  responses  to  the  SIGAR 
recommendations,  DCHA/OTI  agrees  with  the  contents  of  the  report 
subject  to  the  following  clarification  and  corrections. 

1 .  On  page  2,  SIGAR  refers  to  OMB  Circular  A- 122  as  reference  for 
the  definition  of  allowability,  allocability  and  reasonableness  of 
costs.  DCHA/OTI  notes  that  the  cost  principles  outlined  in  OMB 
Circular  A- 122  apply  to  non-profit  organizations.  Since,  DAI  is  a 
for-profit  firm  and  the  award  for  ASI-East  is  a  contract,  we  suggest 
that  SIGAR  cite  Part  3 1  -  Contract  Cost  Principles  and  Procedures 
of  the  Federal  Acquisition  Regulations  (FAR),  more  specifically 
FAR  3 1 .103  -  Contracts  with  Commercial  Organizations. 

2.  On  pages  3  and  5,  SIGAR  references  the  “bum-rate”  approach  to 
program  spending  and  states  that  this  approach  is  “typically  used 
by  other  assistance  programs.”  In  USAID/OTI’s  informal 
technical  comments  provided  to  SIGAR  on  April  9,  2012,  it  was 
clarified  that  the  “bum-rate”  approach  is  not  a  USAID 
programming  approach.  It  was  explained  that  bum-rate  analysis  is 
a  tool  used  by  USAID  for  pipeline  management.  Should  SIGAR 
decide  to  keep  the  terminology  in  the  report,  we  reiterate  our 
request  to  further  explain  the  phrase  “bum-rate”  approach.  As  used 
in  the  report,  it  seems  to  imply  that  USAID  programs  in  general  are 
intended  to  spend  funds  for  the  sake  of  spending  them  fast  without 
regard  for  results. 

3.  On  page  6,  last  paragraph,  SIGAR  states  “DAI  and  OTI 
implemented  program  oversight  and  monitoring  and  evaluation 
systems  through  a  variety  of  means,  including  a  “grants  under 
contract”  approach. . It  should  be  noted  that  “Grants  Under 
Contract”  is  not  a  program  oversight  nor  a  monitoring  and 
evaluation  system.  Rather,  it  is  an  implementation  tool  that  allows 
the  contractor  to  execute  grants  with  non-govemmental 
organizations  (non-profits  or  for-profits). 

4.  On  page  9,  under  the  column  “Description”,  SIGAR  references  the 
Defense  Contract  Audit  Agency’s  (DCAA)  audit  manual  as  basis 
for  citing  DATs  lack  of  oversight  of  its  security  contractor.  We 
request  that  SIGAR  verify  if  the  DCAA  criteria  are  embodied  in 
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the  DAI  contract  standard  provisions,  applicable  USAID  policies 
or  FAR  sections  as  regards  sub-contracts.  If  so,  we  suggest  that 
SIGAR  cite  specific  regulations  or  contract  provisions  that  DAI  did 
not  comply  with  the  oversight  of  its  subcontractor. 


cc:  KYamashita,  USAID/Afghanistan 

CDDEA 
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The  following  are  SIGAR’s  responses  to  USAID  /DCHA/OTFs  letter  dated  June  6,  2012: 


1 .  We  replaced  the  word  “security”  with  “stability”  in  the  report  title. 

2.  Our  draft  report  acknowledged  that  programming  under  the  ASI-East  project  pertained  to  short 
term  stability  programming  and  is  meant  as  a  precursor  to  longer-term  development  programming. 

3.  We  updated  the  report  to  include  the  reference  to  Kunar  and  Wardalc  districts  in  footnote  16. 

4.  We  provided  additional  clarification  of  ASI-East’ s  role  in  the  overall  COIN  strategy  in  footnote  1. 

5.  SIGAR  updated  the  terminology  in  the  report  from  overhead  to  operational  expenses.  However, 
we  segregated  programming  costs  from  all  other  operational  costs  to  highlight  the  amount  of 
expenditures  that  went  directly  toward  activity  level  spending  and  the  amount  that  went  toward 
supporting  those  activities.  As  noted  in  our  report,  the  ASI-East  task  order,  at  the  time  of  award, 
stipulated  an  implied  spending  ratio  of  55  percent  on  program  activities  and  45  percent  on 
operating  costs.  Therefore,  we  do  not  believe  it  is  invalid  to  segregate  these  costs  to  highlight  the 
spending  patterns. 

6.  As  part  of  our  review  of  DAI  and  its  financial  reporting  to  USAID,  we  reviewed  DATs 
timekeeping  and  billing  systems  to  ensure  that  they  were  designed  and  functioning  properly  and 
effectively  with  respect  to  preventing,  detecting,  and  correcting  internal  control  deficiencies. 

Based  on  General  Accounting  Office  Standards  for  Internal  Controls  in  the  Federal  Government, 
dated  November  1999 ,  we  identified  minor  deficiencies  in  DATs  internal  control  that  we  believed 
warranted  USAID’s  attention.  Taking  corrective  actions  now  will  help  mitigate  systemic  or  more 
widespread  occurrences  of  these  or  other  deficiencies.  The  following  lays  out  the  guidance  we 
used  to  address  the  recommendation  regarding  the  time  keeping  and  billing  internal  control 
weaknesses  and  procedural  shortcomings. 

a.  SIGAR  believes  that  increasing  the  frequency  of  formal  supervisory  review  from  monthly  to 
semi-monthly  of  Afghan  timecards  would  help  mitigate  the  potential  of  fraudulent  entry. 
Increasing  supervisory  review  of  timecards  is  based  on  best  practices  and  is  consistent  with 
the  procedures  used  for  ex-patriot  staff. 

b.  Having  complete  and  available  supporting  documentation  attached  to  paid  invoices  is 
required  by  FAR  52.215-2.  Doing  so  helps  facilitate  the  timeliness  and  accuracy  of  invoice 
retrieval  for  questions  regarding  claims,  internal  or  external  audits,  or  other  procedural 
requests. 

c.  SIGAR  maintains  that  DAI  should  have  performed  a  cost  analysis,  once  security  was  in  place, 
to  determine  the  reasonableness  of  the  cost  to  lease  the  items.  This  practice  is  consistent  with 
FAR  31.201-3,  “Determining  Reasonableness.”  A  cost  analysis  would  have  aided  DAI  in 
determining  whether  the  items  it  leased  would  have  been  cheaper  to  purchase  over  the  course 
of  the  project. 

7.  Based  on  M/OAA’s  response  that  it  had  taken  actions  to  determine  that  the  $500,000  questioned 
costs  are  allowable,  we  requested  the  analysis  that  the  DCHA/OTI  Contracting  Officer  used  to 
make  this  determination.  M/OAA  provided  its  response  to  our  request  on  June  20,  2012,  which 
stated  that  it  believed  the  cost  to  be  allowable  based  on  FAR  clause  52.216-7,  Allowable  Cost  and 
Payment  and  FAR  3 1 .201-2,  Determining  Allowability.  In  his  response,  the  Contracting  Officer 
noted  that  (1)  the  costs  were  reasonable  and  in-line  with  other  M&E  studies  conducted  by  Altai 
and  others  in  Afghanistan;  (2)  the  costs  were  allocable  to  the  task  order  and;  (3)  the  terms  of  the 
task  order  and  IQC  were  followed  with  regard  to  subcontracting.  We  do  not  dispute  the 
Contracting  Officer’s  characterization  of  the  reasonableness  of  the  costs;  however,  the  response 
does  not  address  the  lack  of  subcontractor  oversight,  as  discussed  on  page  5  of  the  report.  The 
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mitigation  plan  that  was  implemented  by  OTI  between  DAI  and  Altai  limited  DAI’s  ability  to 
provide  adequate  subcontractor  oversight  as  required  by  FAR  42.202(e)(2).  DAI’s  role  in 
administering  the  subcontract  was  essentially  limited  to  reviewing  and  approving  invoices  for 
payment,  and  did  not  allow  for  normal  oversight  responsibilities.  Therefore,  we  continue  to 
question  whether  the  over  $500,000  was  allowable. 

8.  We  recognize  that  OTI  no  longer  has  ownership  of  the  DSF  and  that  it  is  being  used  by  several 
different  entities.  As  such,  we  updated  Recommendation  4  to  also  include  the  Stabilization  Unit 
Chief,  so  that  OTI  and  the  Stabilization  Unit  may  work  together,  along  with  any  other 
stakeholders,  to  amend  the  DSF.  As  noted  in  our  report,  we  continue  to  believe  that  adding 
project  outcomes  to  level  1  analysis  allows  for  programming  strategy  to  adapt  faster  based  on 
continuous  feedback  instead  of  waiting  for  level  2  impact  analysis  to  be  performed. 

9.  We  recognize  that  the  Stabilization  Unit  has  an  ongoing  effort  with  the  MISTI  program,  which  is 
in  its  beginning  stages,  to  compile  lessons  learned  on  all  programs.  However,  according  to  a 
USAID  Stabilization  Unit  official,  the  MISTI  task  order  would  be  amended  to  remove  the  ASI- 
East  program  from  the  Stabilization  Unit’s  evaluation  of  stabilization  programs  and  the 
Stabilization  Unit  and  OTI  would  rely  on  an  external  contractor,  instead  of  its  own  internal 
assessment,  to  provide  feedback  on  the  ASI-East  program.  Therefore,  we  maintain  that  OTI 
remains  the  appropriate  office  to  produce  a  comprehensive  DSF  lessons  learned  study.  The  DSF 
became  operational  under  OTI  and  remained  the  primary  ASI-East  programming  tool  for  over  2 
years  while  training  over  one  thousand  civilian  and  military  personnel.  Given  the  Stabilization 
Unit’s  role,  since  February  2012,  in  the  ASI-East  program,  we  revised  recommendation  five  to 
also  include  the  Stabilization  Unit  Chief. 

10.  The  reference  to  OMB  Circular  A- 122  was  removed. 

1 1 .  We  believe  the  notion  of  the  burn  rate  approach  provides  important  context  in  light  of  the  method 
used  to  program  the  first  49  grant  activities  by  December  2009  for  ASI-East  under  the  military 
Tactical  Conflict  Assessment  and  Planning  Framework.  The  shift  toward  the  DSF  approach  to 
programming,  as  opposed  to  the  burn  rate  approach,  is  a  more  tactical  approach  and  was  a 
significant  undertaking  on  USAID’s  part  to  steer  away  from  the  use  of  bum  rate  methods  where 
programs  were  slow  to  start  during  their  implementation  phase. 

12.  We  updated  the  report  to  reflect  “Grants  Under  Contract”  as  a  tool. 

13.  The  DCAA  manual  provides  guidance  to  audit  agencies — in  this  case,  to  determine  if  a  contractor 
performed  adequate  oversight  over  a  subcontractor.  FAR  42.202(e)(2)  requires  that  prime 
contractors  provide  oversight  over  their  subcontractors. 
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(This  report  was  conducted  under  the  audit  project  code  SIGAR-047A). 
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SIGAR’s  Mission  The  mission  of  the  Special  Inspector  General  for 

Afghanistan  Reconstruction  (SIGAR)  is  to  enhance 
oversight  of  programs  for  the  reconstruction  of 
Afghanistan  by  conducting  independent  and  objective 
audits,  inspections,  and  investigations  on  the  use  of 
taxpayer  dollars  and  related  funds.  SIGAR  works  to 
provide  accurate  and  balanced  information,  evaluations, 
analysis,  and  recommendations  to  help  the  U.S.  Congress, 
U.S.  agencies,  and  other  decision-makers  to  make 
informed  oversight,  policy,  and  funding  decisions  to 

•  improve  effectiveness  of  the  overall  reconstruction 
strategy  and  its  component  programs; 

•  improve  management  and  accountability  over  funds 
administered  by  U.S.  and  Afghan  agencies  and  their 
contractors; 

•  improve  contracting  and  contract  management 
processes; 

•  prevent  fraud,  waste,  and  abuse;  and 

•  advance  U.S.  interests  in  reconstructing 
Afghanistan. 


Obtaining  Copies  of  SIGAR  To  obtain  copies  of  SIGAR  documents  at  no  cost,  go  to 
Reports  and  Testimonies  SIGAR’s  web  site  (www.sigar.min.  SIGAR  posts  all 

publically  released  reports,  testimonies,  and 
correspondence  on  its  web  site. 


To  Report  Fraud,  Waste,  and  To  help  prevent  fraud,  waste,  and  abuse  by  reporting 
Abuse  in  Afghanistan  allegations  of  fraud,  waste,  abuse,  mismanagement,  and 

Reconstruction  Programs  reprisal  contact  SIGAR’s  hotline: 

•  Web:  www.sigar.mil/fraud 

•  Email:  sigar.pentagon.inv.mbx.hotline@mail.mil 

•  Phone  Afghanistan:  +93  (0)  700-10-7300 

•  Phone  DSN  Afghanistan  318-237-3912  ext.  7303 

•  Phone  International:  +1-866-329-8893 

•  Phone  DSN  International:  312-664-0378 

•  U.S.  fax:  +1-703-601-4065 


Public  Affairs  Public  Affairs  Officer 

Phone:  703-545-5974 

Email:  sigar.pentagon.ccr.mbx.public-affairs@mail.mil 
Mail:  SIGAR  Public  Affairs 
2530  Crystal  Drive 
Arlington,  VA  22202 


